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Objectives 

During  this  tutorial,  we  will  describe 

•  Motivation  and  design  principles  for  an  analysis  course  that 
also  serves  CMMI  and  Six  Sigma 

•  A  problem-solving  methodology  and  its  relationship  to  CMMI 

-  plus,  a  selection  of  its  steps  and  analytical  tools 

•  A  cost  and  schedule  variance  reduction  case  study 

At  the  completion  of  this  tutorial,  you  should  be  able  to  explain 

•  considerations  for  building  your  own  training  courses  (if  you 
need  to  do  so) 

•  an  overview  of  the  DMAIC  methodology 

•  performance  driven  process  improvement 

•  process  variation 

•  Y  to  X  flowdown  (or  “critical  to  quality”  flowdown) 

•  performance  driven  subprocess  selection 

•  how  to  examine  and  improve  the  quality  of  your  data  set 

•  baselining  and  “the  basic  tools” 

•  an  example  of  a  process  performance  models 
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Outline 

O  Context:  The  Value  Proposition 

•  our  (your)  multi-initiative  reality 

•  Six  Sigma  as  a  strategic  enabler 

•  measurement  &  analysis  as  an  integrating  platform 

Approach  to  building  integrated  training 
A  roadmap  for  performance-driven  improvement 
Roadmap  connections  to  CMMI 
Roadmap  execution:  a  case  study  overview 
Roadmap  execution:  demystifying  steps  and  methods 
Summary 
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What  Drives  Process  Improvement? 

Performance  issues:  product,  project — 
and,  eventually,  process  issues 

Regulations  and  mandates 

•  Sarbanes  Oxley 

•  “Level  3”  requirements  to  win  contracts 

Business  issues  and  “burning  platforms” 

•  lost  market  share  or  contracts 

•  continuous  cost  and  cycle  time  improvement 

•  capitalizing  on  new  opportunities 
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There  is  compliance-driven  improvement, 
and  there  is  performance-driven  improvement. 
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Many  Solutions 
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What  solutions  is  your  organization  implementing? 
How  do  they  support  your  organization’s  mission? 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  5 


Carnegie  Mellon 
Software  Engineering  Institute 

Implementation  Considerations 

Many  organizations  are  implementing  one  or  more  models, 
standards,  or  technologies  simultaneously. 

Selection  and  development  considerations  include: 

•  What  is  the  goal? 

•  What  model(s)  or  references  should  be  used? 

•  Should  they  be  implemented  in  parallel  or  sequentially? 

•  Can  they  be  used  “off-the-shelf”  or  is  tailoring  needed? 

•  What  needs  to  be  created  internally? 

Integrated  process  solutions  that  are  seamless  and  transparent 
to  the  engineer  in  the  field  significantly  contribute  to  an 
organization’s  success. 
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CMMI  Staged  and  Six  Sigma 


A  *  Organization-wide  improvements  and  controi 
I  •  Correiation  between  key  process  areas  &  6a  methods 
I  •  6a  used  within  CMM  efforts 


Optimizing 


Process 
improvement 


Q  •  infrastructure  in  piace 
•  Defined  processes  feed  6a 


Quantitativeiy 
Managed 


Procesf  measured 
and  controlled 


U 


Process  characterized  for  the 
organization  and  is  proactive 


•  6a  phiiosophy  &  method  focus 
I  •  6a  “driiidown”  drives  iocai 
(but  threaded)  improvements 


Managed 


Process  characterized  for  projects  and 
is  often  reactive 


6a  may  drive  toward  and 
acceierate  CMMI  solution 


Process  unpredictable  and  poorly  controlled 


Six  Sigma  is  enterprise  wide. 

Six  Sigma  addresses  product  and  process. 

Six  Sigma  focuses  on  “critical  to  quality”  factors. 
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Six  Sigma  and  CMMI  Continuous 

One  possible  approach: 

•  Achieve  high  capability  in  PAs  that  build  Six  Sigma  skills:  MA, 
QPM,  CAR,  OPP 

•  Use  capability  to  help  prioritize  remaining  PAs 
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Tht  Third  lV»/»  • 


Remaining  PAs  ordered  by  business  factors,  improvement  opportunity,  etc.  which  are 
better  understood  using  foundational  capabilities.  CMMI  Staged  groupings  and  DMAIC 
vs.  DMADV  are  also  factors  that  may  drive  the  remaining  order.  [Vickroy  03] 
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A  Different  Tact: 

Six  Sigma  as  Transition  Enabler 

The  SEI  conducted  a  research  project  to  explore  the  feasibility  of 
Six  Sigma  as  a  transition  enabler  for  software  and  systems 
engineering  best  practices. 

Hypothesis 

•  Six  Sigma  used  in  combination  with  other  software,  systems, 
and  IT  improvement  practices  results  in 

-  better  selections  of  improvement  practices  and  projects 

-  accelerated  implementation  of  selected  improvements 

-  more  effective  implementation 

-  more  valid  measurements  of  results  and  success  from  use 
of  the  technology 

Achieving  process  improvement...  better,  faster,  cheaper. 
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Primary  Conclusions 

Six  Sigma  is  feasible  as  an  enabler  of  the  adoption  of  software, 
systems,  and  IT  improvement  models  and  practices  (a.k.a., 
“improvement  technologies”). 

The  CMMI  community  is  more  advanced  in  their  joint  use  of 
CMMI  &  Six  Sigma  than  originally  presumed. 

Noting  that,  for  organizations  studied.  Six  Sigma  adoption  & 
deployment 

•  was  frequently  decided  upon  at  the  enterprise  level,  with 
software,  systems,  and  IT  organizations  following  suit 

•  was  driven  by  senior  management’s  previous  experience 
and/or  a  burning  business  platform 

•  was  consistently  comprehensive. 
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[IR&D  04] 
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Selected  Supporting  Findings  ^ 

Six  Sigma  helps  integrate  multiple  improvement  approaches  to 
create  a  seamless,  single  solution. 

Rollouts  of  process  improvement  by  Six  Sigma  adopters  are 
mission-focused,  flexible,  and  adaptive  to  changing 
organizational  and  technical  situations. 

Six  Sigma  is  frequently  used  as  a  mechanism  to  help 
sustain — and  sometimes  improve — performance  in  the 
midst  of  reorganizations  and  organizational  acquisitions. 

Six  Sigma  adopters  have  a  high  comfort  level  with 
a  variety  of  measurement  and  analysis  methods. 
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Selected  Supporting  Findings  2 

Six  Sigma  can  accelerate  the  transition  of  CMMI. 

•  moving  from  CMMI  Maturity  Level  3  to  5  in  9  months,  or  from 
SW-CMM  Level  1  to  5  in  3  years  (the  typical  move  taking  12- 
18  months  per  level) 

•  underlying  reasons  are  strategic  and  tactical 

When  Six  Sigma  is  used  in  an  enabling,  accelerating,  or 
integrating  capacity  for  improvement  technologies,  adopters 
report  quantitative  performance  benefits  using  measures  they 
know  are  meaningful  for  their  organizations  and  clients.  For 
instance, 

•  ROI  of  3:1  and  higher,  reduced  security  risk,  and  better  cost 
containment 
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CMMI-Specific  Findings 

Six  Sigma  is  effectively  used  at  all  maturity  levels. 

Participants  assert  that  the  frameworks  and  toolkits  of  Six  Sigma 
exemplify  what  CMMI  high  maturity  requires. 

Case  study  organizations  do  not  explicitly  use  Six  Sigma  to  drive 
decisions  about  CMMI  representation,  domain,  variant,  and 
process-area  implementation  order.  However,  participants  agree 
that  this  is  possible  and  practical. 

CMMI-based  organizational  assets  enable  Six  Sigma  project- 
based  learnings  to  be  shared  across  software  and  systems 
organizations,  enabling  a  more  effective  institutionalization  of  Six 
Sigma. 
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IT-Specific  Findings 

High  IT  performers  (development,  deployment,  and  operations) 
are  realizing  the  same  benefits  of  integrated  process  solutions 
and  measurable  results. 

•  However,  they  are  using  the  technologies  and  practices 
specific  to  their  domain  (ITIL,  COBIT,  and  sometimes  CMMI). 

CMMI-specific  findings  apply  to  IT  organizations  who  have 
chosen  to  use  CMMI. 
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Interpretations  &  Moving  Ahead 

There  are  many  potential  benefits  from  thoughtful  and  strategic 
integration  of  multiple  technologies  and  practices. 

Such  integration  is  currently  “state  of  the  art” 

Six  Sigma  and  other  technologies,  can  play  a  role  in  evolving  this 
to  “state  of  the  practice” 

How  do  Process  Improvement  Groups  design  and  rollout 
technologies,  integrated  or  otherwise? 

•  How  does  training,  particularly  “integrated  training,”  support 
their  efforts? 
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A  High  Level  Implementation  Process 


SEI  (or  other  institution) 
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Organization’s  Process  Improvement  Groups: 
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Effective  Transition  Planning 


“Transition”  is  indicated  by  each  of  the  following: 

•  maturation,  introduction,  adoption,  implementation, 
dissemination,  rollout,  deployment,  or  fielding 


Features  of  effective  transition  planning  include: 

•  precision  about  the  problem,  clarity  about  the  solution 

•  transition  goals  and  a  strategy  to  achieve  them 

•  definition  of  all  adopters  and  stakeholders  and  deliberate 
design  of  interactions  among  them 

•  complete  set  of  transition  mechanisms:  a  whole  product 

•  risk  management 

•  either  a  documented  plan  or  extraordinary  leadership 
throughout  the  transition 


[Forrester],  [Schon],  [Gruber] 
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The  “Whole  Product”  Concept 


Economies  of  scale  are 
needed  in  training. 

A  holistic,  “connected” 
approach  is  needed  in 
training. 

Leaving  students  to  their 
own  devices  to  make 
connections  can  be  risky 
and/or  time-consuming. 


[Moore] 

©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  18 


Carnegie  Mellon 

Software  Engineering  Institute 


Training  Chaiienges 

Many  technologies  have  their  own  training. 

•  It’s  not  practical  to  send  everyone  to  all  training  courses. 

•  Yet  it’s  also  not  practical  to  custom-build  all  training. 

Cross  training  (i.e.,  CMMI  &  Six  Sigma) 

•  At  a  strategic  level:  how  to  increase  awareness  so  that 
experts  in  one  technology  can  make  judicious  decisions  about 
adoption  and  implementation  of  another  technology. 

•  At  a  tactical  level:  how  to  balance  the  expertise. 

Who  and  how  many  should  be  trained?  For  instance, 

•  Train  whole  organization  in  internal  process  standards  and 
possibly  basic  Six  Sigma  concepts. 

•  Train  fewer  in  Six  Sigma  BB,  CMMI,  measurement  and 
analysis,  and  other  specialty  areas. 
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Benchmarking 
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Integrated  training  solutions  underway: 

•  DFSS  training  that  includes  awareness  sessions  of 
relevant  technologies 

-  SEI’s  Product  Line  Practices,  ATAM,  CMMI 
engineering  PAs 

•  DFSS  training  that  leverages  ATAM 

•  DMAIC  training  that  references  PSP-based 
instrumented  processes 


Strategic 

Emphasis 


Tactical 

Emphasis 


Our  approach  uses  measurement  &  analysis  as  an  integrator. 
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Outline 

Context:  The  Value  Proposition 
o  Approach  to  building  integrated  training 

A  roadmap  for  performance-driven  improvement 
Roadmap  connections  to  CMMi 
Roadmap  execution:  a  case  study  overview 
Roadmap  execution:  demystifying  steps  and  methods 
Summary 
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Intent  of  New  Analysis  Courses 

Our  task  is  to  build  new  courses  that 

•  Focus  on  analysis 

-  but  more  than  just  SPG 

•  Focus  on  building  skills  hand-on  practice 

•  Support  CMM I 

•  Appeal  to  many  roles 

-  process  improvement  personnel 

-  measurement  personnel 

-  project  team  members 

-  CMM  I  appraisers  (maybe) 

-  Six  Sigma  practitioners 

-  and  so  on 

•  Resonate  with  organizations  at  any  maturity  level 
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A  Base  Architecture 

-  Connecting  all  the  Improvement  Models 


[Kawakita],  [Shiba] 
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Design  Strategy 

Our  Goal  for  the  courses 

•  When  you  return  home,  you  will  have  the  skills  to  tackle 
process  improvement  projects  in  a  way  that  is  informed  by 
your  data. 


Doing  this  requires 

•  a  structured,  but  easily  tailored,  approach 

•  confidence  and  skills  to  use  basic  statistical  methods 

-  sufficient  awareness  of  other  methods  to  know  when 
to  get  help 

•  experience  using  a  real  statistics  package  and  Excel 


SIX  SIGMA  ADVANTAGE 


The  Third  W9/a  * 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  24 


Carnegie  Mellon 

Software  Engineering  Institute 

Considerations 


SIX  SIGMA  ADVANTAGE 

ThB  Third  W9/a  “ 


•  Leverage  other  technologies  and  initiatives. 

-  reuse  demonstrated  frameworks  and  toolkits 

-  build  explicit  connections  to  models 

-  return  to  common  rootsi  don’t  reinvent  the  wheel 

-  define  “certification”  boundaries  and  options 

•  Use  a  project  process  (incl.  design  phases  and  piloting) 

-  Assemble  a  cross-organizational  development  team 

-  Use  Gagne’s  Model  for  instructional  design 

-  Use  Kirkpatrick’s  Four-Level  Evaluation  model 

-  Design  for  fit  with  existing  measurement  courses 


•  Design  for  extensibility:  case  study  approach 

-  allows  easy  swap-in  of  other  domains  and  technologies 

-  al lows  easy  updates  as  core  technologies  evolve 

•  Couple  with  an  annual  Measurement  Practices  Workshop 
(future) 
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Certificates  and  Certifications 


SIX  SIGMA  ADVANTAGE 


The  Third  IV9/S  * 


SEI  Certificate  Programs 
•  analyst  (future) 


Six  Sigma  Practitioner  Certification 

•  SEI  Partners  who  provide  Six  Sigma  training  and  certification 
can  leverage  courses 

-  adjunct,  domain-specific,  Black  Belt  training 

-  domain-specific  Yellow  Belt  training 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  26 


,  ■  Carnegie  Mellon 

Software  Engineering  Institute 

SEMA  M&A  Curriculum 
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Focus  of  Each  Analysis  Course 

I  Process  Measurement  &  Analysis  I 

I  •  Reduce  defects,  waste,  and  cycle  time  by  correcting  special 
^  cause  variation  and  repairing  and/or  improving  processes. 
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Process  Measurement  &  Analysis  II 
•  Prevent  defects  and  ensure  performance  by  using  data  for 
early  detection/correction  of  issues  and  by  optimizing  front- 
end  planning,  requirements,  and  design  processes. 
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Design  Highlights:  Course  Outlines 

Process  Measurement  &  Analysis  I 

•  Introduce  DMAIC  flowchart 

•  Call  Center  Case:  DMAIC  Process 

•  Defect  Containment  Case:  Data  Stratification 

•  Cost  &  Schedule  Case:  Variance  Reduction 

Process  Measurement  &  Analysis  II 

•  Project  Simulation:  Organization  and  Project  Baseline 

•  Defect  Containment  Case: 

-  optimize  inspection,  improve  design  processes 

•  Cost  &  Schedule  Case: 

-  optimize  estimating,  improve  requirements  processes 

•  Project  Process  Optimization  Case: 

-  model  simultaneous  improvements  in  multiple,  interrelated 
processes  via  simulation 
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This  course  uses  case  studies  to  help  you  build  skills. 

•  Call  Center  Case  focus 

-  DMAIC  approach  and  associated  guidance  questions  on 
basic  analytical  methods 

•  Defect  Containment  Case  focus 

-  understanding  variation,  basic  analytical  methods,  and 
SPC  charts 

•  Cost  Schedule  Case  focus 

-  tailoring  the  DMAIC  approach,  understanding  variation, 
excluding  data  properly,  presenting  data  analysis  so  that  it 
is  practical  and  useful;  CMMI  links 


SIX  SIGMA  ADVANTAGE 


The  Third  IV9/S  * 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  30 


Carnegie  Mellon 

Software  Engineering  Institute 


Analysis  1 :  Reinforcing  &  Expanding 
Skills 


Reinforcement 
&  more 
independent 
practice 


Defect  Containment  Case 


Cost  and  Schedule 


DMAIC  Analytical  CMMI 

Approach  Skills  Connections 

- ► 

Increasing  coverage  of  topic 
detail,  added  information 
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Analysis  1 :  Balancing  Lecture  & 
Practice 


Case  1 :  Case  2:  Case  3: 

Call  Defect  Cost  &  Schedule 

Center  Containment  Variance 


Lectures 


Guided 

Exercises 


independent 
&  Group 
Exercises 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  32 


Carnegie  Mellon 
Software  Engineering  Institute 

Outline 

Context:  The  Value  Proposition 
Approach  to  building  integrated  training 
o  A  roadmap  for  performance-driven  improvement 
Roadmap  connections  to  CMMi 
Roadmap  execution:  a  case  study  overview 
Roadmap  execution:  demystifying  steps  and  methods 
Summary 
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A  General  Purpose  Problem-Solving 
Methodology:  DMAIC 


Problem  or  goal  statement  (Y) 


•  Refine  problem  &  goal  .  An  improvement  journey  to  achieve  goals  and  resolve 

statements.  problems  by  discovering  and  understanding 

•  Define  project  scope  &  relationships  between  process  inputs  and  outputs, 

boundaries.  such  as 

Y  =  f(defect  profile,  yield) 

=  f(review  rate,  method,  complexity . ) 
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Codified  method  in  use  across  many  industries 
Domain  specificity 

•  Added  aspects  of  GQ(I)M,  PSM,  and  CMMI  to  make  the 
“domain  independent”  methodologies  more  relevant  to  the 
software  and  system  engineering  domains 

•  Presenting  domain-specific  case  studies 

Can  be  used  to 

•  discover  an  ill-defined  problem  or  opportunity 

•  gather  and  evaluate  information  to  refine  the  problem  or  goal 
statement 

•  gather  and  evaluate  more  information  to  reach  a  proposed 
solution 

•  implement  the  solution 

•  gather  data  to  monitor  success 
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DMAIC  Roadmap 


Define 

i 

— > 

Measure 

i 

I  — > 

Analyze 

Define 

project 

scope 

Identify 

needed 

data 

Explore 

data 

Establish 

formal 

project 

Obtain 
data  set 

Characterize 
process  & 
problem 

Evaluate 
data  quality 

Update 
improvement 
project  scope 
&  scale 

Summarize 
&  baseline 
data 
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Control 

Identify 

possible 

solutions 

Define 

control 

method 

Select 

solution 

Implement 

Document 

Implement 
(pilot  as 
needed) 

Evaluate 
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Establish 

formal 

project 


Define 

— >  I 

Define  ^ 

project 

scope 

What  is  the  current  problem  to  be  solved? 

What  are  the  goals,  improvement  targets,  &  success  criteria? 
What  is  the  business  case,  potential  savings,  or  benefit  that 
will  be  realized  when  the  problem  is  solved? 

Who  are  the  stakeholders?  The  customers? 

What  are  the  relevant  processes,  and  who  owns  them? 


Obtain 
ata  set 


Characterize 

Implement 

Implement 

process  & 

(pilot  as 

Have  stakeholders  agreed  to  the  project  charter  or 
contract? 

cl  What  is  the  project  plan,  including  the  resource  plan  and 
progress  tracking? 

q  How  will  project  progress  be  communicated? 
oasenne - - — - - 


& 
data 
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Measure  Guidance  Questions 


Does  the 

measurement 

system  yield 

accurate, 

precise,  and 

reproducible 

data? 

Are  urgently 
needed 
improvements 
revealed? 

Has  the  risk  of 
proceeding  in 
the  absence  of 
100%  valid 
data  been 
articulated? 


Identify 

needed 

data 

Obtain 
data  set 


Evaluate 
data  quality 


Summarize 
&  baseline 
data 


What  are  the  process  outputs  and  performance 
measures? 

What  are  the  process  inputs? 

What  info  is  needed  to  understand  relationships 
between  inputs  and  outputs?  Among  inputs? 

What  information  is  needed  to  monitor  the  progress  of 
this  improvement  project? 


O^^IULI^I  lO 


lOll 


Is  the  needed  measurement 
infrastructure  in  place? 

Are  the  data  being  collected  and  stored? 


lent 


Document 


What  does  the  data  look  like  upon  initial  assessment? 
Is  it  what  we  expected? 

What  is  the  overall  performance  of  the  process? 

Do  we  have  measures  for  all  significant  factors,  as 
best  we  know  them? 

Are  there  data  to  be  added  to  the  process  map? 

Are  any  urgently  needed  improvements  revealed? 
What  assumptions  have  been  made  about  the 
process  and  data? 
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Analyze  Guidance  Questions 


X 


1 


X 


Are  there  any  hypotheses  that 
need  to  be  tested? 

What  causal  factors  are  driving 
or  limiting  the  capability  of  this 
process? 

What  process  map  updates  are 
needed? 

Are  there  any  immediate  issues 
to  address?  Any  urgent  and 
obvious  needs  for  problem 
containment? 


Explore 

data 


Characterize 
process  and 
problem 


What  do  the  data  look  like? 

What  is  driving  the  variation? 
What  is  the  new  baseline? 

What  are  associated  risks  and 
assumptions  associated  with  the 
revised  data  set  and  baseline? 


solutions 


method 


Evaluate 
data  quality 

Update  ^ 

improvement 
project  scope 
and  scale 

Summarize, 

baseline 

data 

Should  the  improvement  goal  be 
updated? 

Is  additional  data  exploration, 
data  decomposition,  and/or 
process  decomposition  needed? 
Is  additional  data  needed? 

Can  I  take  action?  Are  there 
evident  improvements  and 
corrections  to  make? 

Have  I  updated  the  project 
tracking  and  communication 
mechanisms? 
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•  What  type  of  improvement  is  needed? 

•  What  are  solution  alternatives  to  address  urgent 
issues  and  root  causes  of  identified  problems? 

•  What  are  the  process  factors  to  be  adjusted? 

•  What  is  the  viability  of  each  potential  solution? 

•  What  is  the  projected  impact  or  effect  of  each  viable 
solution? 


improve 

Identify 

possible 

solutions 


What  is  the  action  plan  with  roles,  responsibilities, 
timeline  and  estimated  benefit? 

Is  piloting  needed  prior  to  widespread 
implementation? 


ir  w  1>IIM 


Did  the  solution  yield  the  desired  impact? 

Has  the  goal  been  achieved? 

If  piloted,  are  adjustments  needed  to  the  solution 
prior  to  widespread  rollout?  Is  additional  piloting 
needed? 

How  will  baselines,  dashboards,  and  other 
analyses  change? 
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Select 

solution 


Implement 
(pilot  as 
needed) 


Evaluate 


•  What  are  the 
relative  impacts 
and  benefits? 

•  What  are 
relevant 
technical  and 
logistical 
factors? 

•  What  are 
potential  risks, 
issues,  and 
unintended 
consequences? 
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Define 

project 

scopIT 

Estal  * 

forme 

proje 


I  I  I 

Define 

- > 

M 

•  Should  data  be  compared  to  a  range?  If  so, 
which  range? 

— > 

Control 

Id 

netjuuu 


Does  procedural  adherence  need  to  be 
monitored? 


uaia 


What  updates  are  needed  in  the  measurement  infrastructure? 
What  process  documentation  needs  to  be  updated? 

What  new  processes  or  procedures  need  to  be  established? 
Who  is  the  process  or  measurement  owner  who  will  be  taking 
responsibility  for  maintaining  the  control  scheme? 


Have  we  documented  improvement  projects  for 
verification,  sustainment,  and  organizational  learning? 
What  are  the  realized  benefits? 

Is  the  project  documented  or  archived  in  the  organization 
asset  library? 

Have  documentation  and  responsibility  been  transferred 
to  process  or  measurement  owner? 


Define 

control 

method 


Implement 


Document 
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Define 

Measure 

Analyze 

Improve 

Control 

Benchmark 

Contract/Charter 

Kano  Model 

Voice  of  the 
Customer 

Voice  of  the 
Business 

Quality  Function 
Deployment 

GQIM  and 

Indicator 

Templates 

Data  Collection 
Methods 

Measurement 

System 

Evaluation 

Cause  &  Effect  Design  of 

Diagrams/  Experiments 

Matrix  ..  . 

Modeling 

Failure  Modes  &  . 

Effects  Analysis  ANOVA 

Statistical  Tolerancing 

Inference  Robust  Design 

Reliability  Analysis  Systems 

Root  Cause  Thinking 

Analysis,  inci  5  ^ 

"•’y®  Analysis 

Hypothesis  Test  Perform 

Analysis  Model 

Statistical 

Controls: 

•  Control 

Charts 

•  Time  Series 
methods 

Non-Statistical 

Controls: 

•  Procedural 
adherence 

•  Performance 
Mgmt 

•  Preventive 

measures 

7  Basic  Tools  (Histogram,  Scatter  Plot,  Run  Chart,  Flow  Chart,  Brainstorming,  Pareto  Chart),  Control 
charts  (for  diagnostic  purposes),  Baseline,  Process  Flow  Map,  Project  Management, 

"Management  by  Fact",  Sampling  Techniques,  Survey  Methods,  Defect  Metrics 
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Exercise  /  Discussion 

In  groups  of  3: 

•  Answer  the  “Define  Project  Scope”  questions  for  a  problem  that 
each  of  you  faces  in  your  organization. 

-  What  is  the  current  problem  to  be  solved? 

-  What  are  the  goals,  improvement  targets,  &  success  criteria? 

-  What  is  the  business  case,  potential  savings,  or  benefit  that 
will  be  realized  when  the  problem  is  solved? 

-  Who  are  the  stakeholders?  The  customers? 

-  What  are  the  relevant  processes,  and  who  owns  them? 

•  Discuss  the  “fit”  of  the  DMAIC  roadmap  in  your  organization 

-  How  does  it  fit  with  your  defined  processes? 

-  How  might  it  help  you  define  new  processes? 

-  If  you  are  a  Six  Sigma  organization,  how  does  our  roadmap 
fit  with  your  internal  roadmap? 

Debrief:  Voluntary  Sharing 

TAKE  A  FEW  NOTES.  YOU  WILL  NEED  THEM  LATER. 

©  2005  by  Carnegie  Mellon  University  Version  1 .0  page  43 


SIX  SIGMA  ADVANTAGE 


The  Third  W9/B  * 


. .  ('arnegie  Mellon 

Software  Engineering  Institute 


SIX  SIGMA  ADVANTAGE 

The  Third  W9  /a  " 


Copies  of  Slides 


Later  today:  case  study  and  practice  sessions 


If  you  would  like  copies  of  our  slides, 

add  your  name  to  our  signup  sheet,  or 
send  an  email  to  Debra  Morrison,  dtm@sei.cmu.edu. 
•  Put  “SEPG  Tutorial”  in  the  subject  line. 
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Context:  The  Value  Proposition 
Approach  to  building  integrated  training 
A  roadmap  for  performance-driven  improvement 
e  Roadmap  connections  to  CMMi 

Roadmap  execution:  a  case  study  overview 
Roadmap  execution:  demystifying  steps  and  methods 
Summary 


SIX  SIGMA  ADVANTAGE 


The  Third  W9/a  * 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


page  45 


Carnegie  Mellon 

Software  Engineering  Institute 


SIX  SIGMA  ADVANTAGE 

The  Third  W9/a  “ 


CMMI-SE/SW/IPPD/A  -  Continuous 


CMMI 


•  Organizational  Process 
Focus 

•  Organizational  Process 
Definition 

•  Organizational  Training 

•  Organizational  Process 
Performance 

•  Organizational  Innovation 
and  Deployment 


•  Project  Planning 

•  Project  Monitoring  and 
Control 

•  Supplier  Agreement  Mgmt. 

•  Integrated  Project  Mgmt. 

•  Risk  Management 

•  Quantitative  Project  Mgmt. 


Requirements  Management  •  Configuration  Mgmt. 
Requirements  Development  •  Process  and  Product 


Technical  Solution 
Product  Integration 
Verification 
Validation 


Quality  Assurance 
Measurement  &  Analysis 
Decision  Analysis  and 
Resolution 
Causal  Analysis  and 
Resolution 


IPPD 


Acquisition 


•  Organizational  Environment  •  Supplier  Selection  and  Monitoring 

for  Integration  •  Integrated  Supplier  Management 

•  Integrated  Team  •  Quantitative  Supplier  Management 
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Capability  Evolution  of  Measurement 
via  Generic  Practices 


Generic  Practice 

Focus 

2.8  Monitor  and  control 
the  process 

Monitor  and  control  the  process  against  the  plan  for  performing 
the  process  and  take  appropriate  corrective  action 

3.2  Collect  improvement 
information 

Collect  work  products,  measures,  measurement  results,  and 
improvement  information  derived  from  pianning  and  performing 
the  process  to  support  the  future  use  and  improvement  of  the 
organization’s  processes  and  process  assets 

4.1  Establish  quaiity 
objectives 

Establish  and  maintain  quantitative  objectives  for  the  process 
about  quaiity  and  process  performance  based  on  customer 
needs  and  business  objectives 

4.2  Stabiiize  sub-process 
performance 

Stabiiize  the  performance  of  one  or  more  subprocesses  to 
determine  the  ability  of  the  process  to  achieve  the  estabiished 
quantitative  quaiity  and  process  performance  objectives 

5.1  Ensure  continuous 
process  improvement 

Ensure  continuous  improvement  of  the  process  in  fuifiiiing  the 
reievant  business  objectives  of  the  organization 

5.2  Correct  common 
cause  of  probiems 

Identify  and  correct  the  root  causes  of  defects  and  other 
probiems  in  the  process 
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DMAIC  Relationship  to  CMMI  PAs 

Overall 

•  DMAIC:  a  problem  solving  approach 

•  CMMI:  a  process  &  measurement  deployment  approach 

PAs  that  align  with  DMAIC  include  the  following: 

•  MA,  GPs 

•  QPM,  CAR,  OID  (either  “continuous”  or  high-maturity  view) 

A  DMAIC  project  may  leverage  these  existing  processes: 

•  PP,  PMC,  IPM 

•  OPP  for  organization  level  execution,  mgmt,  oversight 

PAs  through  which  DMAIC  may  be  incorporated  into 
organizational  process  definition  include  the  following: 

•  OPF,  OPD 
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[Penn  &  Siviy  05] 
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Relationship  to  CMMI  PAs 

PAs  “eligible”  for  DMAIC-based  improvement 

•  all 

PAs  with  links  to  the  analytical  toolkit  include 

•  Decision  Analysis  &  Resolution 

-  e.g.,  concept  selection  methods,  such  as  Pugh’s 

•  Risk  Management 

-  e.g.,  Failure  Modes  &  Effects  Analysis  (FMEA) 

•  Technical  Solution 

-  e.g..  Design  FMEA,  Pugh’s 


[Penn  &  Siviy  04] 
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“Define”  Roadmap  Steps 

•  Define  project  scope  o  Align  process  improvements  with 
business  objectives 

-  Organization  Process  Focus  (SG  1 ) 

-  Organization  Process  Performance  (SG  1 ) 

-  GP4.1,  GP5.1 

•  Establish  formal  project  o  Establish  improvement  projects 

-  Organization  Process  Focus  (SG  1 ) 

-  Organization  Innovation  and  Deployment  (SG  1) 

-  Implied  by  GP  4.1 ,  GP  5.1 
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“Measure”  and  “Analyze”  Roadmap  Steps 

•  Define  data  and  establish  repositories 

-  Measurement  and  Analysis  (SG  1) 

-  Organization  Process  Definition  (SG  1) 

-  Organization  Process  Performance  (SG  1) 

-  Causal  Analysis  and  Resolution  (SG  2) 

-  Quantitative  Project  Management  (SG  2) 

-  GP2.2,  GP3.2,  GP5.1 

•  Baseline  data 

-  Organizational  Process  Performance  (SG  1) 

•  Analyze  data 

-  Measurement  and  Analysis  (SG  2) 

-  Organization  Process  Performance  (SG  1) 

-  Causal  Analysis  and  Resolution  (SG  1) 

-  GP2.8,  GP5.2 
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Relationship  to  CMMI  Goals  &  Practices 

“Improve”  and  “Control”  Roadmap  Steps 

•  Identify  Improvement  Alternatives 

-  Decision  Analysis  and  Resolution  (SG  1) 

-  Organization  Innovation  and  Deployment  (SG  1) 

-  Organization  Process  Performance  (SG  1 ) 

-  GP5.1 

•  Control  Processes 

-  Measurement  and  Analysis  (SG  2) 

-  Organization  Process  Performance  (SG  1) 

-  Organization  Innovation  and  Deployment  (SG  2) 

-  Causal  Analysis  and  Resolution  (SG  2) 

-  Quantitative  Project  Management  (SG  2) 

-  GP2.8,  GP4.2 
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Discussion  /  Exercise 

Our  lists  are  not  exhaustive. 

In  your  group  of  3 

•  Are  there  other  dimensions  via  which  DMAIC  and  CMMI  are 
connected? 

•  Are  there  other  goals  which  link  to  DMAIC? 

-  Do  any  specific  practices  come  to  mind? 

•  Are  there  any  other 

•  Are  there  other  analytical  methods  from  the  Six  Sigma  toolkit 
that  are  frequently  used  within  a  Process  Area? 

•  How  are  CMMI  and  DMAIC  different?  Synergistic? 

Debrief:  Voluntary  sharing 
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Context:  The  Value  Proposition 
Approach  to  building  integrated  training 
A  roadmap  for  performance-driven  improvement 
Roadmap  connections  to  CMMi 
o  Roadmap  execution:  a  case  study  overview 

Roadmap  execution:  demystifying  steps  and  methods 
Summary 
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What  is  the  cost  and  schedule  performance  of  one  of  the  projects 
in  your  organization?  Of  your  organization’s  projects  as  a 
whole? 


Does  your  answer  reflect 

•  total  variance  from  initial  estimate?  Or  variance  from  most 
recent  re-estimates? 

•  all  projects?  Or  just  certain  types  of  projects? 

•  “level  of  effort”  or  maintenance  work? 


We  are  going  to  “run  through”  a  high  level  view  of 
the  cost-schedule  variance  reduction  case  from 
the  course. 

Keep  track  of  any  methods  or  approaches  that 
you  don’t  understand.  We  will  ask  in  --I  hour. 
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Organizational  Context 


Motivation  for  project 

•  innprove  customer  satisfaction 

-  indicated  by  field  defects  and  effort  and  schedule  variance 

Project  portfolio 

•  both  development  and  maintenance 

•  size  and  complexity  vary 

•  schedules  from  <1  month  to  >18  months 

CMMI  implementation 

•  assessed  at  SW-CMM  Level  3  five  years  ago 

•  began  transition  to  CMMI  four  years  ago 

•  Working  toward  high  maturity 

•  striving  to  implement  new  Process  Areas  to  add  value,  not 
just  achieve  compliance 
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Partial  list  of  work  that  is  already  complete 

•  measurement  infrastructure  for  project  and 
product  metrics 

•  measurement  infrastructure  akin  to  GP2.8  and 
GP3.2  for  processes  implemented  per  SW-CMM 

•  SPG  pilots  within  selected  organizational  units 

•  initial  data  rollups  for  management 

•  initial  benefits  calculations  (ROI  and  other 
indicators) 
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Excerpt  of  CMMI  Plans 


What  “M&A”  type  efforts  lie  ahead,  to  fulfill  CMMI-related  goals? 

•  ensure  compliance  to  MA  Process  Area 

•  organizational  and  project  baselines  (OPP  and  QPM  SPs) 

•  model  building  at  the  project  level  (QPM  SPs) 

•  relating  process  behavior  and  performance  to  product  and 
service  quality  (OPP  and  QPM  SPs) 

•  quantitatively  manage  key  processes  and  subprocesses 
(QPM  SPs) 

Problematic  CMMI  language  and  practices 

•  “process  performance  model”  (OPP  SPs) 

•  subprocess  selection  (QPM  SPs) 

If  you  are  working  on  Level  2  &  3  PAs,  don’t  tune  out. . . 

•  Remember  our  research  project  findings:  the  lessons  from 


this  case  can  be  applied  to  build  a  foundation  for  high  matt 
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Documented  M&A  Process 


0 


Select 
Business  Goal 


ective 


ce  Thresholds 

- ► 


1 


No  “Issues” 


Gather 

Data 


Identified 

Thresholds 


•Project  Performan^ 
•Measures  Quality 
•SPI  Implementation 


Improvements 


Draft  Improvement 
•Snapshot  (1®*  Iteration  ->  Baseline)  Goal  (SMART)  or 
•Issues  (Validity  of  data,  Quality  of  Identify  focus  area 
Data,  Variance  (performance) 


Identify 

Gather  Data/Analyze  Data 

Perform  Causal 

Prioritize 

Identify 

Possible 

- ► 

Analysis 

w 

^\ctual 

p 

Potential 

Causes 

rx  A^^uses 

Solutions 

(Brainstorm) 

V _ k 

1®’  Iteration  ^  Final  Goal 


Goal  Refinement 


Develop 

Implement 

Action  Plan 

Improvement 
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Customer  Satisfaction  Goal  Structure 


Carnegie  Mellon 

Software  Engineering  Institute 


SIX  SIGMA  ADVANTAGE 
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Cost/Schedule  Measurement  System 


Monthly  effort,  schedule  (earned  value  based) 

•  variance:  (current  actual  -  most  recent  estimate) 

-  compared  to  variance  specification  (+/-  20%) 

-  text  entry  for  out  of  specification  explanations 

•  data  manually  transferred  from  project  files  to  monthly  report 

•  cost  in  terms  of  effort  hours 

•  schedule  performance  derived  from  effort  hours 

Completed  project  data:  a  special  data  collection 

•  variance:  (original  plan  -  final  actual) 

-  categorized  by  internal  and  external  causes 

-  categorized  by  life-cycle  phase 

•  cost  in  terms  of  effort  +  purchase  costs 

•  schedule  in  terms  of  calendar  days 

•  each  replan  recorded 
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What  is  Monthly  Average  &  Variability? 

Cost/Schedule  Performance  Baseline 


All  Org  Units,  all  projects, 
October  to  March 


•Reminder:  This  is  (current 
actual  -  most  recent  estimate) 

•Averages  within  spec,  and 
close  to  0 


•Need  to  examine  extreme 
values,  especially  for  cost 


•  even  if  extreme  values  are 


outliers,  it  looks  like  we  need 

Range:to  investigate  variability 

60% 


mean — ^ 


|jciueiiiiie 

^75'^  percentile 

Median:  50'^  percentile 


■^25'^  percentile 
10'^  percentile 

page  62 


Carnegie  Mellon 

Software  Engineering  Institute 


SIX  SIGMA  ADVANTAGE 

The  Third  W9/a  “ 


Disposition  of  Extreme  Vaiues 


Extreme  values  investigated 

•  removed  all  documentation  projects 

•  removed  small  project  that  was  re-baselined  26  times 

Immediate  improvement  opportunity 

•  establish  data  validation  guidelines 


We’ll  talk  more  about 
measurement  system 
evaluation  and  outliers 
this  afternoon 
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What  is  Monthly  Average  &  Variability? 

Cost/Schedule  Performance  Baseline,  Outliers  Removed 


100-r 

80- 

60- 


I  20- 


-40 -. 


-60- 


s 


I 


Even  with  outliers  removed, 
variance  has  more  variability 
than  expected. 

There  is  month  to  month 
consistency. 


-80- 

-100-1 - 1 - 1 - 1 - 1 - 1— 

1:  oct-01  2:  nov-01  3:  dec-01  4:  jan-02  5:  feb-02  6: 

month  label 

Scaled  to  match  schedule  variance  chart, 
some  values  below  -100 


This  and  following  slides 
consonant  with  Six  Sigma  toolkit, 
CMMI  MA,  QPM,  OPP,  CAR 
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Overall  Monthly  Performance  Stats 

Cost/Schedule  Performance  Baseline,  Outliers  Removed 


In-process  effort/cost  data 
•  All  life-cycle  phases,  all  projects,  Oct  -  June 


cost 

schedule 

average 

-2 

-7.6 

standard  deviation 

25 

19.2 

n 

770 

770 

%  outside  spec 

17% 

17% 

Reminder:  variance  =  current  actual  -  most  recent  estimate 
Observations 

•  averages  running  close  to  target 

•  more  data  than  expected  outside  of  spec 

•  high  variability 
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How  Do  Projects  Typically  End? 

Cost/Schedule  Performance  Baseline,  Outliers  Removed 


%  effort  variance 

%  sched  variance 

avg 

C^1 3%^ 

std  dev 

36^ 

median 

min  to  max 

C7128%  toTT^ 

capability  notes 

C!43.8^ 

CT  39^ 

(spec  =  +/-  20%) 

outside  spec 

outside  spec 

Reminder:  This  is  the  total  cumulative  variance. 

•  (initial  plan  -  final  actual) 

•  customer-driven  changes  are  included 


Some  extreme  values  still  present 

•  there  are  no  valid  reasons  to  remove  or 
segment  the  data 

Large  %  of  data  outside  specification 

•  process  not  capable 


] 

■ 

im 

LSL 

s 

/ 
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Cost/Schedule  Data  Quality 


Carnegie  Mellon 

Software  Engineering  Institute 


®®@0© 


Accuracy 

•  re-baselining  project  estimates  biases  the  data,  promotes  the 
perception  that  performance  is  better  than  it  actually  is 

Repeatability 

•  different  estimating  methods  across  life  cycle,  across  projects 

•  unclear  definition  of  “project” 

Completeness 

•  Which  project  types  are  “in”  and  which  are  “out”? 

•  sparse  data  -  some  parts  of  organization  better  represented 

•  completed  project  data  missing  from  organization  data 

Sampling  Homogeneity 

•  All  monthly  data  was  being  rolled  together,  regardless  of 
which  part  of  life  cycle  it  represented 
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Initial  Baseline  Summary  (All  Data) 


Existing  customer  satisfaction  information  was  positive. 

Cost/Schedule  data  frequently  and  consistently  “out  of  spec,” 
both  monthly  and  final  data. 

•  But,  the  average  looked  great! 

There  were  few  field  defects  and  the  inspection  process  seems 
to  be  effective. 

Many  measurement  infrastructure  improvements  were  identified 

•  customer  surveys 

•  improved  operational  definitions 

•  improved  automation 

-  to  improve  quality  and  minimize  quantity  of  missing  data 

•  add  completed  project  data  into  central  repository 

•  group  data  by  life  cycle  phase  or  project  %complete  for 
reporting 
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Why  Pursue  Cost  &  Schedule 
Variation  Reduction? 


Why  should  we  care  about  effort  &  schedule  variance  reduction? 

•  our  customers  are  pretty  happy 

•  our  project  managers  seem  to  have  things  under  control 


What  are  the  business  drivers?  What  is  the  benefit  for  the  effort 
that  will  likely  be  invested? 

•  competitive  advantage 

•  funding  increasingly  harder  to  obtain 

•  credibility 

•  domino  effect  of  projects  running  late 

•  more  small  projects  anticipated 

•  fewer  “level  of  effort”  projects  anticipated 

•  operational  cost-based  budget 

Why  does  YOUR  organization  care  about  cost  &  schedule 
performance? 
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While  the  re-alignment  may  not  change  the  specific  improvement 
activities,  it  will  positively  affect  “change  management”  efforts. 
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Cost  and  schedule  variance  and  measurement  quality  selected 
as  primary  improvement  opportunities 

Dashboard  established  to  monitor  unintended  consequences 
while  organizational  focus  shifts  to  cost  &  schedule 
improvements 
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Segmenting  the  Data: 

Are  there  Differences  by  Org  Unit? 

Schedule  Variance,  all  projects,  Oct  01  to  Jun  02 

Visual  indicator  of 
significant  difference 

Circle  size  influenced 
by  sample  size 

Concentric  circles 
indicate  no  difference. 

Separate  circles 
indicate  difference. 

Overlapping  circles 
are  somewhere  in 
between. 


Are  there  statistically  significant  group-to-group  differences:  NO 
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Variance  Causal  Analysis 


Brainstorming  Workshop 

•  process  and  measurement  points  of  contact  met  for  2  days  to 
review  data,  brainstorm  about  sources  of  variation 

Transformed  original  brainstorm  list 

•  initial  experiential  assessment  of  frequency,  impact  of  each 
cause  code 

•  refined  operational  definitions  and  regrouped  brainstorm  list 

•  tagged  causes  to  historical  data 

•  refined  again 

Process  decomposition 

•  decomposed  process  into  four  main  subprocesses 

•  mapped  cause  codes  to  process 

•  identified  cause  codes  that  are  resolved  in-process 
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Transformed  original  brainstorm  list 

•  initial  experiential  assessment  of  frequency,  impact  of  each 
cause  code 

•  refined  operational  definitions  and  regrouped  brainstorm  list 

•  tagged  causes  to  historical  data 

•  refined  again 


Final  list  included  such  things  as 

•  missed  requirements 

•  underestimated  task 

•  over  commitment  of  personnel 

•  skills  mismatch 

•  tools  unavailable 

•  EV  method  problem 

•  planned  work  not  performed 

•  external 


Coded  historical  data 
by  cause  codes. 

Evaluated  for 
frequency,  severity  of 
impact. 
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Co-Optimized  Pareto  Anaiysis 


Impact 
#  (from 
Pareto) 

Schedule 

Effort 

Organization 
Slice  1 
Schedule 

Organization 
Slice  1  Effort 

Organization 
Slice  2 
Schedule 

Organization 
Slice  2  Effort 

1 

Under¬ 

estimated 

Task 

Tools 

Under¬ 
estimated  Task 

Under¬ 

estimated 

Task 

Tools 

Tools 

2 

Tools 

Assets  not 
available 

EV  Problems 

Under 

planned 

rework 

Skills 

mismatch 

Under¬ 

estimated 

Task 

3 

EV  Problems 

Under 

planned 

rework 

Missed 

requirements 

Missed 

requirements 

Under¬ 

estimated 

Task 

Missed 

requirements 

4 

Missed 

requirements 

Planned 
work  not 
performed 

Under  planned 
rework 

EV  Problems 

Missed 

Requirements 

Unexpected 
departure  of 
personnel 

5 

Skills 

Mismatch 

Under¬ 

estimated 

task 

Asset 

availability 

Planned  work 
not 

performed 

Unexpected 

departure 

EV  Problems 
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SMART*  Schedule  Variance  Goal 


Reduce  the  total  variance  by  decreasing  the  variance  of  the  top  3 
internal  causes  by  50%  in  1  year. 

Reduce  the  impact  of  external  causes  by  50%. 

Indicators  (defined  via  GQIM  Indicator  Template): 

•  trend  for  each  cause  independently 

•  trend  for  total  variance 


Goa!  statements  address  what  needs  to  be 

*Make  your  goais  SMART: 

changed,  scope  of  the  organization,  current 

Specific 

and  target  measure  and  timeiine: 

Measurabie 

To  [increase/decrease]  primary  metric 

Attainabie  or  Agreed-upon 

[where?]  [from  existing  ievei]  to  [target] 

Relevant  or  Realistic 

by  [when?] 

Timely  or  Timebound  J 
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Cause  Code:  Underestimated  tasks 
Process:  Project  management 

Subprocesses:  Planning 

•  establish  requirements 

•  define  project  process 

•  perform  detailed  planning 


CMMI 

Friendly 


Requirements  Management  y 


As  subprocesses  are  explored,  process  mapping  and 
input/output  analysis  may  be  used  with  (or  based  on) 
ETVX  diagrams. 


Six 

Sigma 

Friendly 
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Schedule  Variance  Root  Cause 


Root  causes  of  common  cause  variation 

•  inexperience  in  estimation  process 

•  flawed  resource  allocation 

•  estimator  inexperience  in  product  (system) 

•  requirements  not  understood 


Root  causes  of  special  cause  variation 

•  too  much  multitasking 

•  budget  issues 
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Management  by  Fact _ 

®  Problem  /  Goal  Statement 

Reduce  the  total  schedule  variance  by  decreasing  the  variance  of 
the  top  3  internal  causes  by  50%  in  1  year. 


variance  w/ 


(mTm/u 

an  comoa 

ris< 

8 

Tc  1' 

'  2  M0?.,h  ^ 

Variance  for  top  3  causes: 

•  underestimated  tasks 

•  EV  method  problem 

•  missed  requirements 


Mo^th 


Prioritization  & 

Counter  Measures 

Impact,  Capability 

Root  Cause 

First:  Gather  real  time  data  and 

In  total,  these  countermeasures 

1.  Inexperience 

verify  “data  archaeology” 

will  remove  15%  of  typical 

2.  Resource  allocation 

Then: 

variance. 

3.  Requirements  not 

1 . 

(as  possible,  list  impact  of  each 

understood 

2.... 

countermeasure) 

4 . 

- (  A  ICAR)- 

(  1  XcMloiDt 
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From  Organizational  to  Project  View*: 
Variability  Across  the  Life  Cycle 

It  was  hypothesized  that  there  are  more  cost  &  schedule 
variabilities  early  in  the  project  than  later. 

•  relative  to  most  current  estimate 
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wider  limits 
for  projects 
in  planning 
phase 


project  cost  index 

1  0  n - 


Month 


narrower  limits 
for  projects  in 
execution  phase 


*As  a  reminder:  For  those  working  on  Level  2  and  3,  the  project  view  can  be  addressed  as  part  of  MA 
and  the  GPs  for  respective  process  areas.  This  can  build  a  foundation  for  higher  maturity. 
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Within-Project  Variance  Results 

Three  groupings  (practical  and  statistical  basis): 

•  <20%  complete  (planning) 

•  20-80%  complete  (majority  of  effort;  routine  execution) 

•  >80%  complete  (converging  on  completion) 

More  data  anomalies  to  resolve 

•  When  *exactly*  do  projects  start  and  stop? 

•  Are  data  from  on-hold  projects  included?  (No.) 

•  Should  level  of  effort  data  be  included?  (Yes,  but  separately.) 
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EV  Estimate-At-Completion  Model 

A  transformation  from  traditional  control  chart  view  to  traditional  PM  view 


Limits  for  schedule  and  cost  variance  for  each  grouping  projected 
to  a  range  around  the  estimate-at-completion  values. 


Future  enhancements:  sensitivity  anaiysis  and  prediction  modei 


©  2005  by  Carnegie  Mellon  University 


page  82 


Version  1.0 


Carnegie  Mellon 

Software  Engineering  Institute 


DMAIC  Summary  for  Full  Case 

Multiple  D-M-A  iterations 

•  iteration  1 :  problem  identification  &  project  selection 

-  Reduce  cost  and  schedule  variance 

-  Improve  data  quality 

•  subsequent  iterations: 

-  “peel  the  onion”  to  better  understand  problems 

-  establish  specific  quantitative  goals 

Improvements  instituted 

•  measurement  infrastructure  expansion,  automation 

•  cost  and  schedule  variance  cause  code  taxonomy 

•  estimating  (training,  minor  process  adjustments) 

•  adoption  of  “management  by  fact”  (MBF)  format 

•  homogeneous  sampling  for  cost  and  schedule  data 
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Monitoring  &  control  mechanisms 

•  organization:  dashboards  with  charts  for  cost,  schedule, 
defects,  data  quality,  and  customer  satisfaction 

•  projects:  cost/schedule  prediction  model 
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Methodology/Model  Connections 

WV  Model  (the  ‘base  architecture’): 

•  DMAIC  used  for  project  identification 

CMMI 

•  Process  areas*  used:  MA,  OPP,  QPM,  OID,  CAR 

•  Process  areas  touched:  PP,  PMC,  RD,  REQM 

•  Terms  addressed:  Baseline,  process  performance  model 

Measurement  best  practices 

•  indicator  template  key  component  of  measurement  plan 
Six  Sigma 

•  problem-solving  approach  influenced  design  and  definition  of 
measurement  &  analysis  processes 

•  used  MBF  as  an  organizational  innovation 

•  indicator  templates  added  as  a  domain-specific  tool  to  the  Six 
Sigma  toolkit 
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*See  Addenda  for  list  of  CMMI  Process  Areas 
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Skills-Building  for  the  Full  Case 

In-class  skills-building  (practice)  sessions 

•  baselining,  using  box  plots,  distributions,  other  charts 

•  Hypothesis  testing,  means  comparison  tests 

•  Prioritizing  causes  for  action 

•  Failure  Modes  Effects  Analysis 

In-class  discussions  and  other  exercises 

•  What  drives  cost  &  schedule  variance 

•  risks  of  using  historical  data 

•  small-sample  sizes  and  homogeneous  sampling 

•  corrective  action  guidance  (as  part  of  indicator  template) 


SIX  SIGMA  ADVANTAGE 


The  Third  W9/a  * 


page  85 


©  2005  by  Carnegie  Mellon  University 


Version  1.0 


^  Carnegie  Mellon 

Software  Engineering  Institute 


SIX  SIGMA  ADVANTAGE 


The  Third  W9/a  * 


N 

After  lunch,  we  will 
•  decompose  various  elements^f  the  case 


Practice 


If  you  would  like  copies  of  our  slides, 

Add  your  name  to  our  signup  sheet,  or 

send  an  email  to  Debra  Morrison,  dtm@sei.cmu.edu. 

•  Put  “SEPG  Tutorial”  in  the  subject  line. 
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Outline 

Context:  The  Value  Proposition 
Approach  to  building  integrated  training 
A  roadmap  for  performance-driven  improvement 
Roadmap  connections  to  CMMi 
Roadmap  execution:  a  case  study  overview 
o  Roadmap  execution:  demystifying  steps  and  methods 
Summary 
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Our  Approach  for  the  Afternoon 


We  excerpted/shortened  several  lectures  from  the  course 

•  Statistical  thinking 

•  “Define”:  business  benefits 

•  “Measure”: 

-  distilling  data:  segmentation,  input/output  analysis,  Y  to  x 

-  Baselining 

-  Measurement  system  analysis 

•  “Analyze”: 

-  Exploring  the  data 

-  Characterizing  the  process 

-  Root  Cause  Analysis 

We’ve  interspersed  the  lectures  with  exercises,  discussions  and 
examples 

•  The  exercises  will  be  based  on  the  cost-schedule  case  just  shown,  or 
on  the  DMAIC  project  scope  you  defined  earlier  today 

•  Examples  are  drawn  from  any  of  the  3  cases  in  the  course 
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Internal  View 

Fitness  to  a 

Standard 

Compliance 


Fitness  to 

Cost 

and  Speed 

Lean  Efficiency 
Rapid  Delivery 


Customer  View 


Fitness  for 

Use 

Performance 


Fitness  to 

Latent 

Requirement 


Customer  Intimacy 
Innovation 

Competitive  Advantage 
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Six  Sigma  and  ‘Lean’  -  Natural  Partners 


Six  Sigma 


‘Lean’ 


/ 


/ 


/ 


^  •  Customer  Focus: 

•  Defects 

•  Business  Growth 

•  Performance 

•  Thinking  Statisticaliy 

•  Process  ‘Control’ 


\ 


\ 


\ 


/ 


\ 


/ 


/ 


\ 


\ 


/ 

/ 


\ 


\ 

I _ Powerful  in _ 

I  Concert  i 


\ 


\ 


\ 


N. 


Systematic  Development  \ 
of  Infrastructure: 

•  ‘Beits,’  ‘Champions,’ 
Leadership  Engagement 


\ 


\ 


/ 


Speed 
Flow 

Removing  Constraints 

•  Delays 

•  Value-add  analysis 

•  Reducing  Waste 
Reducing  Complexity 
Increasing  Flexibility 


\ 


\ 


\ 


\ 


\ 


/ 


/ 


/ 


/ 


DMAIC:  Follow  the  Defects 


Lean:  Follow  the  Time 
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Lean  Origins  and  Evolution 


Seminal  work  at  Toyota  laid  ‘Lean’  foundations: 

•  Understanding  and  reducing  ‘work  in  process’  (WIP) 

-  Highly  visible  in  manufacturing 

materials-waiting  scrap  inventory 

-  Present,  but  often  less  visible,  in  software  and  IT 

decisions-waiting  rework  multiple  projects  underway 

•  Making  process  flow,  timing,  and  costs  visible, 
so  dynamics  and  constraints  can  be  understood 
and  exploited. 

Extension  to  other  processes  has  been  a  natural  from  the  beginning 

-  with  history  of  success  in  services 

-  emerging  success  stories  in  software  and  IT 
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Everything  is  a  process. 

All  processes  have  inherent  variability. 

Data  are  used  to  understand  variation  and  to  drive  decisions  to 
improve  the  processes. 
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Data  Spread  due  to 
Conrmon  Cause  Variation 


[ASQ  00],  [ASA  01] 


/ 


Special  Cause  Variation 


Original  Mean 


New  mean  after  improvement 

(Spread  due  to  ODrrrnDn  cause 
N^riation  vvill  re-establish  itself.) 
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Process  Off  Taraet 


Excessive  Variation 


Center 

Process 


Tar 

get 

V 

LSL  USL 

Reduce 

Spread 
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Types  of  Data 


Attribute 


(a.k.a.,  nominal 
categorical, 
digital) 


Increasing 

Information 

Content 


Placing  observations  into  categories 


Ordinal  Attribute  data, 

with  >  or  <  relationships 
among  the  categories 

WL<WL<WL 

ABC 


Examples 

Defect  counts  by  type 
Job  titles 


Satisfaction  ratings: 
unsatisfied,  neutral,  delighted 

Risk  estimates:  low,  med,  high 

CMM  maturity  levels 


Variables 


(a.k.a.,  measures, 
continuous, 
analog) 


Assignment  of 
observations  to  points  on 
a  scale ...  enabling 
determination  of  interval 
sizes  and  differences 


Interval 


Ratio 


deg.  F,  C 


Time 

Cost 

Code  size 
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(5) 

A  sample  is  a  set  of  observations  selected  from  a  population. 


A  statistic  is  a  function  average  -  +  Xg  +  X3 

that  summarizes  observations. 


A  sampling  distribution  maps 
values  of  a  sample  statistic  (x)  vs. 
their  probability  of  occurrence  (y). 


3 


A  population  consists  of  the 
totality  of  observations  with 
which  we  are  concerned.  ^ 
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Descriptive  Statistics 


Measures  of  Central  Tendency,  location,  middle,  the  balance  point 


Mean  -  ^  measured  values 

10486.3  

(the  center  of 
gravity  by  value) 

(the  average) 

= - =  227.9 

46 

Median  =  the  midpoint  by  count 

Mode  =  the  most  frequently  observed  point 
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Measures  of  Dispersion:  Spread,  variation,  distance  from  central  tendency 


Range 

maximum  -  minimum 

2 

S(Xi-A) 

V  ariance  =  — - 

n 

StandardDeviatioi^  V  Variance 

Average  squared  distance  from  the 
population  mean 

a 

In  the  units  of  the  original  measures; 
indicator  of  the  spread  of  points  from 
the  mean 
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aN  the  data  ? 

No,  a  sample 


Inferential 

Statistics 


Statistics 

Mean,  median,  mode, 
variance  (a^),  standard 
deviation  (a),  range,  etc. 

Estimates  of  mean,  median, 
mode,  variance  (s^), 
standard  deviation(s),  etc. 

Issues 

Measurement  system 
accuracy  and  repeatability 

Measurement  system 
accuracy  and  repeatability 

Sampling  bias 

Choice  of  statistic 

Greek  letters  (e.g.  a,  p) 
usually  designate  descriptive  stats 

Roman  letters  (e.g.  s,  y) 

usually  designate  inferential  (sample)  stats 

Surveys  returned  from 
about  120  customers 


Example 


L^u  vvt;  I  idvt 


Descriptive 

Statistics 


All  of  last  quarter’s  calls 
to  the  support  hotline 
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Define  Guidance  Questions 


Define 
project 
scope 

Establish 
formal 
project 


•  What  is  the  current  problem  to  be  solved? 

rs  the  goals,  improvement  targcL,  <lx  oULLL3i5-u:ii^a? 
What  is  the  business  case,  potential  savings,  or  benefit  tl^ 
lealized  when  the  problem  is  solved? 

Who  are  tne  stakefidld^l’y  ^  I  htJ  CUiilomerT^ 

What  are  the  relevant  processes  and  who  owns  them? 


L^UUUI  I  It^l  IL 


Have  stakeholders  agreed  to  the  project  charter  or 
contract? 

What  is  the  project  plan,  including  the  resource  plan  and 
progress  tracking? 

How  will  the  project  progress  be  communicated? 
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Reconciling  Different  “Voices” 

Voice  of  the  Business  (VOB) 

•  term  to  describe  the  stated  and  unstated  needs  or 
requirements  of  the  business/shareholders 

Voice  of  the  Customer  (VOC) 

•  term  to  describe  the  stated  and  unstated  needs  or 
requirements  of  the  customer 

Voice  of  the  Process  (VOP) 

•  term  to  describe  the  performance,  capability  of  the 
organization’s  processes 

Understanding  the  relationship  between  these  voices 

•  enables  appropriate  selection  of  processes  to  begin  study 

•  further  refines  project  scope  and  enables  the  establishment  of 
a  formal  project 

•  lays  the  foundation  for  work  to  be  done  in  “measure,  analyze, 
improve” 
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Evaluating  Business  Benefits 


Things  to  consider: 

•  What  is  the  business  motivation  for  this  project? 

-  How  big  is  the  problem? 

-  What  are  the  implications? 

-  What  is  the  projected  payback  or  net  value? 

•  Why  now? 

•  What  would  be  the  consequences  if  we  delayed  or  failed  to 
execute? 

•  How  does  this  project  fit  with  others? 

-  relative  priority 

-  leverage/support  re:  other  initiatives 
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Measuring  Benefits 

$$  saved 

Return  on  Investment 
Return  on  Equity 
Cash  Flow  Rate  of  Return 
Cost  of  Poor  Quality 
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Cost  of  Poor  Quality  (COPQ) 


Costs  of  Poor  Quality  (COPQ) 

Measured 

(in  some  businesses) 


Long  Cycle  Times 
Lost  Sales/Customen 


Contract  Penaities 

Write  Offs  Support  Costs 


Audit  Cost 


Requirements  Errors 
Rework 

^Employee  Attrition 


Time  Value  of  Money 

Lost  Customer  Loyalty  Cash  Misapplication  Working  on  the  Wrong  Thing 
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Evolving  the  COPQ  Indicator 


ld0ntifiC8tion!  (based  on  a  financial  metric) 

Rep  Supervisor  Storage/Monitoring 

+ 

Classification:  $$ 

Hard  vs.  soft  savings 
Appraisal  cost  vs.  cost  avoidance 


e.g.  Y  =  unit  cost  per  call  = 


+ 


+  ?? 


Validation: 


Confidence 


Define 

Initial  (1)  COPQ 

Analyze 

COPQ 

Improve 

Pre-Implementation  Cost-Benefit  Analysis 

Control 

Post-Implementation  Cost-Benefit  Analysis 

Realization 

Verified  (longer  term)  Cost-Benefit  Analysis 

Moderate 


High 
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Validating  Business  Costs  &  Benefits 


While  you  can  estimate  some  non-value-added 
costs,  it  is  important  to  validate  costs  with  your 
organization’s  accounting  department  early  in  the  ( 
project. 


This  will: 

•  confirm  the  business  case 

•  document  the  project’s  “before”  picture  as  a  baseline 

•  document  the  project’s  net  financial  gains 
(‘after’  -  project  costs  -  “before”) 
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Example:  Call  Center  Case 
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Call  Center  Project  Scope 


Problem  Statement  (from  benchmarking  and  high  level  baselining): 
Competitors  are  growing  their  levels  of  satisfaction  with  support 
customers,  and  they  are  growing  their  businesses  while  reducing  support 
costs  per  call.  Our  support  costs  per  call  have  been  level  or  rising  over  the 
past  18  months,  and  our  customer  satisfaction  ratings  at  or  below 
average.  Unless  we  stop,  or  better,  reverse  this  trend  we  are  likely  to  see 
compounded  business  erosion  over  the  next  18  months. 

Business  Case: 

Increasing  our  new  business  growth  from  1%  to  4%  (or  better)  would 
increase  our  gross  revenues  by  about  $3mm.  If  we  can  do  this  without 
increasing  our  support  costs  per  call,  we  should  be  able  to  realize  a  net  gain 
of  at  least  $2mm. 

Goal  Statement: 

Increase  the  call  center’s  industry-measured  Customer  Satisfaction  rating 
from  its  current  level  (75%)  to  the  target  level  (85%)  without  increasing 
support  costs,  by  end  of  Q4. 
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Measure  Guidance  Questions 


Define 

Measure 

•  Does  the 

measurement 
system  yield 
accurate, 
precise,  and 
reproducible 
data? 

Are  urgently 
needed 
improvements 
revealed? 

Has  the  risk  of 
proceeding  in 
the  absence  of 
100%  valid 
data  been 

I? 

MA  lOPD 


Obtain 
data  set 


Evaluate 
data  quality 


What  are  the  process  outputs  and  performance 
measures? 

What  are  the  process  inputs? 

What  info  is  needed  to  understand  relationships 
between  inputs  and  outputs?  Among  inputs? 

What  information  is  needed  to  monitor  the  progress  of 
this  improvement  project? 


O^IULI^^I  lO 


I  lOll  IVJKJ 


Is  the  needed  measurement 
infrastructure  in  place? 

Are  the  data  being  collected  and  stored? 


lent 


Document 


What  does  the  data  look  like  upon  initial  assessment? 
Is  it  what  we  expected? 

What  is  the  overall  performance  of  the  process? 

Do  we  have  measures  for  all  significant  factors,  as 
best  we  know  them? 

Are  there  data  to  be  added  to  the  process  map? 

Are  any  urgently  needed  improvements  revealed? 
What  assumptions  have  been  made  about  the 
process  and  data? 
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Distilling  Data 

Raw  Data 

Measure 


Events 


Measure 

& 

Analyze 


Graphical 

Analysis 

What?  When?  Where? 
How?  How  much? 


TIME  SERIES 

^/'Vv 


HISTOGRAM 


lIL 


PARETO  CHART 


▼ 

Patterns 


Analyze 


Causal 

Analysis 

Why? 

What’s  really  going  on? 


What’s  driving  the  patterns? 


STATISTICAL 
Tests  and  Models 


y 

Structure 


TRANSFER  FUNCTIONS 
and  OPTIMIZATION 
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Funneling”  to  the  Critical  x’s 


Define 


Charter,  Y 


Establish  project,  process,  and  requirements  scope 
This  identifies  Y(s)  and  the  “search  space”  for  x’s 

◄ - ► 


All  the  x’s 


Improve 

Control 


Modeling 

Piloting 


Control  Plan 


Measure 

r  1  UOcob  IVIdpb  ^ — 

Y-to-xTree 

I/O  Factor  Assess  ^  10-15 

Initial  C&E 

FMEA 

Analyze 

/  T 

Graphical  AnalysiX.  \  Q  1  n  / 

(simple)  Stats  >  \  O  -1 U  / 

Data-Driven  C&E/  \  / 

Initial  focus 


4-8 


3-6 


Analysis  narrows 
the  focus 


Critical  x’s  confirmed 


X’s  to  Control 
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Methods  to  Narrow  Our  Focus 


What  are  the  process  outputs  (y’s)  that  drive  performance? 
What  are  key  process  inputs  (x’s)  that  drive  outputs  (process 
performance)  and  overall  performance? 

^  ■  .  .ions 


•  cause  &  effect  diagrams 

•  cause  &  effect  matrices 

•  failure  modes  &  effects  analysis 

Using  these  techniques  yields  a  list  of  relevant,  hypothesized, 
process  factors  to  measure  and  evaluate. 
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Description 

A  logical  reasoning  about  which  data  groupings  have  different 
performance,  often  verified  by  basic  descriptive  statistics. 

Procedure 

•Consider  what  factors,  groupings,  segments,  and  situations 
may  be  driving  the  mean  performance  and  the  variation  in  Y. 
•  Draw  a  vertical  tree  diagram,  continually  reconsidering  this 
question  to  a  degree  of  detail  that  makes  sense. 

•Calculate  basic  descriptive  statistics,  where  available  and 
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Segmentation  vs.  Stratification 

Segmentation 

•  grouping  the  data  according  to  one  of  the  data  elements  (e.g., 
day  of  week,  call  type,  region,  etc.) 

•  gives  discrete  categories 

•  in  general  we  focus  on  the  largest,  most  expensive, 
best/worst  -  guides  “where  to  look” 

Stratification 

•  grouping  the  data  according  to  the  value  range  of  one  of  the 
data  elements  (e.g.,  all  records  for  days  with  “high”  volume  vs. 
all  records  with  “low”  volume  days) 

•  choice  of  ranges  is  a  matter  of  judgment 

•  enables  comparison  of  attributes  associated  with  “high”  and 
“low”  groups  -  what’s  different  about  these  groups? 

•  guides  diagnosis 
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Input  /  Output  Analysis 


Supplier 


inputs 

Process 

Outputs 

Step  1 

outputs 

II  ipULO 

- ^ 

inputJ 

- ► 

Step  2 

- ►  Outputs 

Customer 


Assess  the  Inputs: 

•Controllable:  can  be  changed  to  see  effect  on  key  outputs 
(also  called  “knob”  variables) 

•Critical:  statistically  shown  to  have  impact  on  key  outputs 
•  Noise:  impact  key  outputs,  but  difficult  to  control 
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Controlled  factors  are  within  the  project  team’s  scope  of 
authority  and  are  accessed  during  the  course  of  this  project. 

Studying  their  influence  may  inform: 

•  cause-and-effect  work  during  Analyze 

•  solution  work  during  improve 

•  monitor  and  control  work  during  Control 


Uncontrolled  factors  are  factors  we  do  not  or  cannot  control. 


We  need  to  acknowledge  their  presence  and,  if 
necessary,  characterize  their  influence  on  Y. 

A  robust  process  is  insensitive  to  the  influence 
of  uncontrollable  factors. 
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Example:  Development  Process  Map 


Design 


O  Requirements 
*  Estimate 
O  Concept  design 


Code 

Compile 

Unit 

Test 

g 

O  Resources 

O  Code 

O  Executable  Code 

♦  Code  Stds 

•  Test  Plan,  Technique 

♦  LOC  counter 

•  Operational  Profiles 

Y 

f  Interruptions 

> 

< 

> 

Y 

> 


•  Detailed  Design 

•  Test  cases 

•  Complexity 

•  Data:  Design  Review 
defects,  Fix  time, 
Phase  duration 


•  Code 

•  Data:  Defects, 
Fix  time.  Defect 
Injection  Phase, 
Phase  duration 


•  Executable 
Code 

•  Data:  Defects, 
Fix  time.  Defect 
Injection  Phase, 
Phase  duration 


•  Functional 
Code 

•  Data:  Defects, 
Fix  time.  Defect 
Injection  Phase, 
Phase  duration 


Inspection 

Critical  Inputs 

♦  Standard  Procedure 

Q  Rework 

V 

t 

Noise 

•  Control  Knobs 

J 
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Lean  Methods  and  Tools 


Process  Mapping 

•  (Customer)Value-Add  /  Non  Value-Add  /  Business  Value-Add 

•  Cycle  time  and  throughput  analysis  (see  “Little’s  Law) 


Process  Dynamics  /  Constraints  Analysis 

•  Characterizing  bottlenecks,  queues 

•  Identifying  throughput,  capacity,  and  quality  constraints 
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Mapping  Variation:  Vaiue  Map 

Identify  the  process  to  map. 

Identify  the  boundaries. 

Create  input-process-output  for  the  critical  processes. 

Create  the  process  map. 

Color  code  each  step  identifying  value. 

•  green  =  value  added 

•  red  =  non  value  added 

•  yellow  =  non  value  added  but  necessary 

Identify  hand-off  points,  queues,  storage,  and  rework  loops  in  the 
process. 

Quantitatively  measure  the  map  (throughput,  cycle  time,  and 
cost). 

Validate  map  with  process  owners. 
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Flowdown  (Y-to-x)  Tree  -1 

Description 

•  Y  to  X  is  a  hypothesis  tool 

-  Depicts  hypothesized  causal  relationships  between 
customer-critical  performance  measures  and  process 
factors 

-  represents  portions  of  a  transfer  function  over  time 
Procedure 

•  Draw  a  vertical  tree  diagram  to  depict  the  causal  relationship. 

-  Use  information  from  process  mapping,  natural 
segmentation  as  inputs. 

•  Identify  x's  as  uncontrollable  vs.  controllable  and  measurable. 

•  Select  y’s  and  x's  for  initial  data  collection  and  evaluation. 
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Flowdown  (Y-to-x)  Tree  -2 

Considerations 

•  Flowdown  trees  are  very  useful  when  Y  is  hard  to  measure 
directly  or  hard  to  influence. 

•  Cause  &  effect  diagrams  are  another  means  of  diagramming 
hypothesized  causal  relationships. 

•  Subsequent  “measure”  and  “analyze”  tasks  will  help 
determine  the  strength  and  the  nature  of  each  important  x-Y 
relationship. 

•  The  initial  selection  of  y’s  and  x's  for  data  collection  may  be 
based  on  logic  and  data  availability.  As  more  about  the 
process  is  understood,  quantitative  causal  relationships  will 
drive  selections. 
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Tool  Connections 


FMEA 


Begin  to 
understand 
x’s  impact  on 
Y’s;  further 
prioritize  x’s. 


Function  or 
Process  Step 

Potential  Failure 
Mode 

Potential  Failure 
Effects 

S 

E 

V 

Potential  Causes 

0 

C 

C 

Detection  Provisions 

D 

E 

T 

R 

P 

N 

0 

0 

0 

0 
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“Finding  X” 


Input  Output  Analysis 


Classify 

x's. 


Inputs 


Diitpiits 


Cause  and  Effects  Matrix 


o 

o 

3 

O 

aT 

a; 

0 

X 

ifi 


Prioritize 
steps 
and  x's. 

1  1 
i  i 

1  1 
i  i 

1  1 
i  i 

1 

i 

P^ocessSlep 

Process  ,npu. 

10 

Deep 
dive  into 
process 
steps 


Refined  Process  Map 


Understand 

prioritized 

process 

steps. 
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Y  to  y  Example:  Cost  Schedule  Case 


Achieve 

Customer 

Satisfaction 


Success  Indicators 
(Customer  Survey) 


Y’s 


_ 

L 

>  < 

Delivered 

Product 

Quality 

{other  goals} 

I 


Plot,  plot,  plot 
defect  and  reliability  data: 

•  trends 

•  distributions 

•  control  charts  (c-charts) 

•  scatter  plots  ^ - 


I 


In-process  quality 

y’s 

Analysis  Indicators 

Cost  and 
Schedule 

Variance 

'  Plot,  plot,  plot 
cost  and  schedule  data: 

•  trends 

•  distributions 

•  control  charts  (x,  mr) 

•  scatter  plots 


Progress 

Indicators 


-i 
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Example:  Cost-Schedule  Case 
Segmentation  by  Organizational  Unit 

Schedule  Variance,  all  projects,  Oct  01  to  Jun  02 

Visual  indicator  of 
significant  difference 

Circle  size  influenced 
by  sample  size 

Concentric  circles 
indicate  no  difference. 

Separate  circles 
indicate  difference. 

Overlapping  circles 
are  somewhere  in 
between. 

Are  there  statistically  significant  group-to-group  differences:  NO 
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Example:  Call  Center  Customer  Sat. 
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Example:  Call  Center  Support  Cost 
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=  identified  as  factors  for  first  round  data  collection 
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Exercise:  Segmentation  &  I/O  Analysis 

Earlier  today,  you  answered  the  “define  project  scope”  questions 
for  a  problem  you  are  facing 

Identify  a  key  performance  measure  (a  ‘big  Y’) 

Since  that  measure  cannot  be  changed  directly  -  begin  to  build  a 
‘Y  to  x’  tree  to  identify  factors  (x’s),  groups,  situations,  or  other 
categorical  reasons  that  may  be  driving  performance  or 
variation? 

For  some  key  ‘x’s  -  classify  whether  they  would  be  ‘controllable’ 
or  ‘uncontrolled’  in  the  scope  of  the  project  at  hand. 

Think  ahead  about: 

-  Availability  of  data  to  study  x-Y  relationships 

-  Quality  of  the  measurement  system  for  x  data 
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Measure  Guidance  Questions 


Does  the 

measurement 

system  yield 

accurate, 

precise,  and 

reproducible 

data? 

Are  urgently 
needed 
improvements 
revealed? 

Has  the  risk  of 
proceeding  in 
the  absence  of 
1 00%  valid 
data  been 
articulated? 


Measure 


Identify 

< 

needed 

data 

Obtain 
data  set 


Summarize 
&  baseline 
data 


What  are  the  process  outputs  and  performance 
measures? 

What  are  the  process  inputs? 

What  info  is  needed  to  understand  relationships 
between  inputs  and  outputs?  Among  inputs? 

What  information  is  needed  to  monitor  the  progress  of 
this  improvement  project? 


What  does  the  data  look  upon  initial  assessment?  Is  it 
what  we  expected? 

What  is  the  overall  performance  of  the  process? 

Do  we  have  measures  for  all  significant  factors,  as 
best  we  know  them? 

Are  there  data  to  be  added  to  the  process  map? 

Are  any  urgently  needed  improvements  revealed? 
What  assumptions  have  been  made  about  the 
process  and  data? 


Is  the  needed  measurement  infrastructure 
in  place? 

Are  the  data  being  collected  and  stored? 
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Measurement  System  Analysis  (MSA) 

Purpose 

•  Understand  the  data  source  and  the  reliability  of  the 
process  that  created  it 

Indicators  of  data  quaiity  and  reiiabiiity 

•  Vaiidity 

•  integrity 

•  Accuracy 

•  Repeatability 

•  Reproducibiiity 

•  Stabiiity 

•  credibiiity 
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MSA  Tools  &  Methods 

Sometimes,  a  simple  “eyeball”  test  reveals  problems. 

More  frequently,  a  methodical  approach  is  warranted. 

Useful  tools  and  methods  include 

•  process  mapping 

•  indicator  templates 

•  operational  definitions 

•  initial  evaluation/exploration  assessment  using  statistical  tools 

•  checklists 
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Use  common  sense,  basic  tools,  and  good  powers  of 
observation. 
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MSA  Practical  Tips 

Frequently  encountered  problems  include  the  following: 

•  wrong  data 

•  missing  data 

•  skewed  or  biased  data 

Map  the  data  collection  process. 

•  Know  the  assumptions  associated  with  the  data. 

Look  at  indicators  as  well  as  raw  measures. 

•  ratios  of  bad  data  still  equal  bad  data 

Data  systems  to  focus  on  include  the  following: 

•  manually  collected  or  transferred  data 

•  categorical  data 

•  startup  of  automated  systems 
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Exercise:  What  if  I  Skip  MSA? 


What  if... 

•  All  O’s  in  the  inspection  database  are  really  missing  data? 

•  “Unhappy”  customers  are  not  surveyed? 

•  Delphi  estimates  are  done  only  by  experienced  engineers? 

•  A  program  adjusts  the  definition  of  “line  of  code”  and  doesn’t 
mention  it? 

•  Inspection  data  doesn’t  include  time  and  defects  prior  to  the 
inspection  meeting? 

•  Most  effort  data  are  tagged  to  the  first  work  breakdown 
structure  item  on  the  system  dropdown  menu? 

•  The  data  logger  goes  down  for  system  maintenance  in  the 
first  month  of  every  fiscal  year? 

•  A  “logic  error”  to  one  engineer  is  a  “ _ ”  to  another 


Which  are  issues  of  validity?  Bias?  Integrity?  accuracy? 
How  might  they  affect  your  conclusions  and  decisions? 
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Measure  Guidance  Questions 


Define 


Does  the 

measurement 

system  yield 

accurate, 

precise,  and 

reproducible 

data? 

Are  urgently 
needed 
improvements 
revealed? 

Has  the  risk  of 
proceeding  in 
the  absence  of 
100%  valid 
data  been 
articulated? 

MA 


Identify 

needed 

data 

Obtain 
data  set 


Evaluate 
data  quality 


What  are  the  process  outputs  and  performance 
measures? 

What  are  the  process  inputs? 

What  info  is  needed  to  understand  relationships 
between  inputs  and  outputs?  Among  inputs? 

What  information  is  needed  to  monitor  the  progress  of 
this  improvement  project? 


O^IULI^^I  lO 


lOLI 


Is  the  needed  measurement 
infrastructure  in  place? 

Are  the  data  being  collected  and  stored? 


lent 


Document 


What  does  the  data  look  like  upon  initial  assessment? 
Is  it  what  we  expected? 

What  is  the  overall  performance  of  the  process? 

Do  we  have  measures  for  all  significant  factors,  as 
best  we  know  them? 

Are  there  data  to  be  added  to  the  process  map? 

Are  any  urgently  needed  improvements  revealed? 
What  assumptions  have  been  made  about  the 
process  and  data? 
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Baselining 

What  is  baselining? 

•  establishing  a  snapshot  of  performance  and/or  the 
characteristics  of  a  process 

Why  baseline  performance? 

•  provides  a  basis  by  which  to  measure  improvement 

How  is  it  done? 

•  map  the  process  of  interest 

-  including  scope  (process  boundaries)  and  timeframe 

•  gather  data 

-  sample  appropriately 

•  summarize  data  using  basic  tools 
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The  7  Basic  Tools 

Description 

•  Fundamental  data  plotting  and  diagramming  tools 

-  cause  &  effect  diagram 

-  histogram 

-  scatter  plot 

-  run  chart 

-  flowchart 

-  brainstorming 

-  Pareto  chart 

•  The  list  varies  with  source.  Alternatives  include  the  following: 

-  statistical  process  control  charts 

-  descriptive  statistics  (mean,  median,  etc.) 

-  check  sheets 
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7  Basic  Tools:  Cause  &  Effect 
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7  Basic  Tools:  Chart  Examples  2 


50C 

40G 

30C 

20c 


10c 


Development  Effort  (person 
months)  vs.  Size  (KSLOC) 


0  10  20  30  40  50  60  70 

Size  (thousands  of  executable  source  lines) 

■Q 


Histogram 


0 

JD 

E 

3 


10 

& 

& 

4- 

a 


Scatter  Plot 


34  36  38  40  42  44  46  48  50  52  54  56 

Product-Service  Staff  Hours 
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90  T - 

80  I 

70 

^60 

■■£50 

«40 

0  30 
20 
10 

0  i-l— 
20 


Defects  Removed  By  Type 


Pareto  Chart 


40  80  50  10 

Defect  Type  Codes 


Run  Chart 


Mean  Time  To  Repair 
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7  Basic  Tools:  Chart  Examples 


Box  &  whisker  plot 
for  assessment  data 


mean 


^  > 

f 

90*h  percentile 
75*h  percentile 
-Median:  50*'^  percentile 
'25*h  percentile 
1 percentile 


CfJM  BENCHMARK 
SEI  Level  3 
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7  Basic  Tools:  Chart  Examples 

SPC  Chart:  Individual,  Moving  Range 


_C^ 

03 

"D 

■> 

ID 

C 


60 


50 


40 


30 


20 


10 


Zone  C 

LCL 


10  15  20 

Subgroup  No. 
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Example:  Cost/Schedule  Monthly 
Performance  Baseline 


All  Org  Units,  all  projects, 
October  to  March 

•Reminder:  This  is  (current 
actual  -  most  recent  estimate) 

•Averages  within  spec,  and 
close  to  0 

•Need  to  examine  extreme 
values,  especially  for  cost 

•  even  if  extreme  values  are 

Range: 

60%  outliers,  it  looks  like  we  need 
to  investigate  variability 
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Exercise:  Outliers 

What  is  an  outlier? 

•  a  data  point  which  does  not  appear  to  follow  the  characteristic 
distribution  of  the  rest  of  the  data 

•  an  observation  that  lies  an  abnormal  distance  from  other 
values  in  a  random  sample  from  a  population 


Consider  this  cost  variance  data: 

_ ^99 

•  • 

-  1 3,  22,  1 6,  20,  1 6,  1 8,  27,  25,  30(333, 

40 

-  average  =  50.9,  standard  deviatiorS.-9d 

1.9 

If  “333”  is  a  typo  and  should  have  been  “33” 

-  corrected  average  =  23.6,  corrected  standard  deviation  =  8.3 

But,  what  if  it’s  a  real  value? 
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In  groups  of  3 

•  share  your  approach  for  deciding  if  and  when  to  remove 
extreme  values  from  data  sets 


[Frost  03],  [stats-online] 
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Removing  Outliers 

There  is  not  a  widely-accepted  automated  approach  to  removing 
outliers. 

Approaches 

•  Visual 

-  examine  distributions,  trend  charts,  SPC  charts,  scatter 
plots,  box  plots 

-  couple  with  knowledge  of  data  and  process 

•  Quantitative  methods 

-  interquartile  range 

-  Grubbs’  test 

Time  is  running  short. . .  So,  we  shall  leave  it  to  the  student  to  look  up  the 
quantitative  methods  using  the  listed  references  as  “homework.  ”  (We  will 
display  1  slide  on  interquartile  range  during  break) 
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[Frost  03],  [stats-online] 
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When  Not  to  Remove  Outliers 


When  you  don’t  understand  the  process 

Because  you  “don’t  like  the  data  points”  or  they  make  your 
analysis  more  complicated. 

Because  IQR  or  Grubbs  method  “says  so” 


When  they  indicate  a  “second  population” 

•  identify  the  distinguishing  factor  and  separate  the  data 


When  you  have  very  few  data  points 


Innocent  until  proven  guilty 
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Copies  of  Slides 


After  break: 

Exploring  &  Characterizing  Data 


If  you  would  like  copies  of  our  slides, 

add  your  name  to  our  signup  sheet,  or 
send  an  email  to  Debra  Morrison,  dtm@sei.cmu.edu. 
•  Put  “SEPG  Tutorial”  in  the  subject  line. 
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Interquartile  Range:  Example 


Interquartile  Range 


30-16  =  14 


Procedure 

1 .  Determine  1  and  3'^'^  quartiles 
of  data  set:  Q1 ,  Q3 

2.  Calculate  the  difference: 
interquartile  range  or  IQR 

3.  Lower  outlier  boundary  = 

Q1  -1.5*IQR 

4.  Upper  outlier  boundary  = 

Q3  +  1.5*IQR 


0 

Upper  outlier  boundary 
30  +  1.5*14  =  51 


0 

Lower  outlier  boundary 
16- 1.5*14  = -5 


Example  adapted  from  “Metrics,  Measurements,  &  Mathematical  Mayhem,”  Alison  Frost,  Raytheon,  SEPG  2003 
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Analyze  Guidance  Questions 


Are  there  any  hypotheses  that 
need  to  be  tested? 

What  causal  factors  are  driving 
or  limiting  the  capability  of  this 
process? 

What  process  map  updates  are 
needed? 

Are  there  any  immediate  issues 
to  address?  Any  urgent  and 
obvious  needs  for  probiem 
containment? 


Evaluate 
data  quality 


Summarize, 

baseline 

data 


Update 
improvement 
project  scope 
and  scale 


What  do  the  data  look  like? 
What  is  driving  the  variation? 
What  is  the  new  baseline? 
What  are  associated  risks  and 
assumptions  associated  with 
revised  data  set  and  baseline? 


Should  the  improvement  goal  be 
updated? 

Is  additional  data  exploration, 
data  decomposition,  and/or 
process  decomposition  needed? 
Is  additional  data  needed? 

Can  I  take  action?  Are  there 
evident  improvements  and 
corrections  to  make? 

Have  I  updated  the  project 
tracking  and  communication 
mechanisms? 
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Exploring  the  Data 

Considerations  when  answering  the  guidance  questions 

•  What  should  the  data  look  like?  And,  does  it? 

-  first  principles,  heuristics  or  relationships 

-  mental  model  of  process  (refer  to  that  black  box) 

-  what  do  we  expect,  in  terms  of  cause  &  effect 

•  Are  there  yet-unexplained  patterns  or  variation?  If  so, 

-  conduct  more  Y  to  x  analysis 

-  plot,  plot,  plot  using  the  basic  tools 

•  Are  there  hypothesized  x’s  that  can  be  removed  from  the  list? 
The  objective 

•  Go  into  “characterize”  with  as  close  as  possible  to  the  “right 
list”  of  Y’s,  y’s,  and  x’s  to  study/characterize  in  a  detailed  way. 

•  (At  the  end  of  characterize,  the  list  is  further  narrowed  and 
prioritized) 
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‘Lean’  Formulas  for  Data  Exploration 
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Lead  Time  = 

Work  ‘Units’  in  Process 

Average  Completion  Rate 

Process  Efficiency  = 


Value-Added  Process  Time 
Total  Process  Time 


Cycle  Time 
Performance  Ratio  = 


Actual  Time 
Theoretical  Time 
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A  typical  exploration  question: 
Are  There  Multiple  Populations? 


Multimodal  distributions 
point  to  multiple 
processes. 


O 


When  there  are  multiple  populations, 

•Do  we  understand  the  causes  based  on  work  done  in  “measure”? 
•Do  we  need  to  further  explore  Y  to  x  relationships? 

•Do  we  need  to  segment  or  stratify  the  data  for  further  analysis? 
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Characterizing  Your  Process 

What  causal  factors  are  driving  or  limiting  the  capability  of  this 
process? 

•  Which  x’s  are  or  are  not  significant? 

•  Can  plausible  changes  in  x’s  deliver  targeted/desired  changes 
in  y’s  and  Y’s? 

•  Do  we  need  to  find  more  x’s? 

•  Do  we  need  to  refine  goals? 

To  support  the  ability  to  answer  these  questions,  are  there  any 
hypotheses  that  need  to  be  tested? 

•  How  do  we  test?  (tests  for  significant  difference,  correlations, 
experiments) 

What  is  the  stability  and  capability  of  the  process? 

•  What  are  assignable  causes  for  “special  cause”  variation? 

•  What  are  root  causes  for  “common  cause”  variation? 
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Testing  for  Differences 

Comparing  a  process  or  product  to  “specification” 

•  Is  the  process  on  aim? 

•  Is  the  variability  satisfactory? 

Comparing  two  processes  or  products  or  populations 

•  Are  the  means  (or  medians)  the  same? 

•  Is  the  variation  the  same? 

Approaches  to  determining  if  means/medians  are  the  same 

•  one-way  analysis  of  variance  (ANOVA) 

•  means  comparison  tests 

•  confidence  interval  for  the  delta,  \Xq  - 
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More  Advanced  Tools  that  Support 
Characterization 


Y 


Attribute 

Variable 

0 
-1— « 

Pareto 

Scatter  Plot 

Chi-Square 

Time  Series 

-1— » 

< 

Contingency  Table 

Analysis  of  Variance  (ANOVA) 

0 

Logistic  Regression 

Scatter  Plot 

0 

Correlation 

0 

> 

Linear  and  Non-linear  Regression 

F-test 
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Scatterplot  of  Defects  vs  Complexity  (McCabe) 


Complexity  (McCabe) 


May  indicate ... 
but  not  prove  causation 


Pearson  correlation  of  Complexity 
(McCabe)  and  Defects  =  0.837 


P-Value  =  0.000 


I  30. 


25- 


20- 


Scatterplot  of  Defects  vs  if-then 


•  • 


•  •  • 


•  • 
•  • 


150  175  200 


225  250 

if-then 


Pearson 
correlation  of 
if-then  and 
Defects  =  0.930 

P-Value  =  0.000 
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Variability  &  Capability 


Process 

Capability 

vs. 

Capable 

Process 


Voice  of  the  Process 
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Example:  Final  Project  Cost  &  Schedule 

Cost/Schedule  Performance  Baseline,  Outliers  Removed 


%  effort  variance 

%  sched  variance 

avg 

-2% 

13% 

std  dev 

33% 

36% 

median 

2% 

7% 

min  to  max 

-95%  to  50% 

-128%  to  71% 

capability  notes 

C43.8^ 

d39^ 

(spec  =  +/-  20%) 

outside  spec 

outside  spec 

Reminder:  This  is  the  total  cumulative  variance. 

•  (initial  plan  -  final  actual) 

•  customer-driven  changes  are  included 


Some  extreme  values  still  present 

•  there  are  no  valid  reasons  to  remove  or 
segment  the  data 

Large  %  of  data  outside  specification 

•  process  not  capable 


] 

■ 

im 

LSL 

s 

/ 
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Analyze  Guidance  Questions 


Are  there  any  hypotheses  that 
tn  hn  tn-^nrl'^ — 

(hat  causal  factors  are  drivTr 
or  limiting  the  capability  of  this 
i;ocess? 

WhaTprobiitiS  map  updates  are 
needed? 

Are  there  any  immediate  issues 
to  address?  Any  urgent  and 
obvious  needs  for  problem 
containment? 


Explore 

data 


Characterize 
process  and 
problem 


What  do  the  data  look  like? 
What  is  driving  the  variation? 
What  is  the  new  baseline? 
What  are  associated  risks  and 
assumptions  associated  with 
revised  data  set  and  baseline? 


Evaluate 
data  quality 

Update  ^ — 

improvement 
project  scope 
and  scale 

Summarize, 

baseline 

data 
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Should  the  improvement  goal  be 
updated? 

Is  additional  data  exploration, 
data  decomposition,  and/or 
process  decomposition  needed? 
Is  additional  data  needed? 

Can  I  take  action?  Are  there 
evident  improvements  and 
corrections  to  make? 

Have  I  updated  the  project 
tracking  and  communication 
mechanisms? 
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“Real”  Process  Characteristics 

Real  processes  exist  or  occur  only  in  execution/operation  J 

Exactly  stable  conditions  never  exist  with  real  processes^ 

Real  processes  are  never  entirely  free  of  perturbations  or 
anomalies.^ 

Real  processes  are  subject  to  entropy,  requiring  continual  repair 
of  its  effects. 

Variations  in  the  “system  of  causes”  results  in  process  variation. 
Process  variation  fluctuates  over  time. 
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^  Deming,  W. Edwards.  Out  of  the  Crisis.  Cambridge,  Mass.:  Massachusetts  Institute  of  Technology,  Center  for  Advanced  Engineering,  1986. 
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Process  Analysis 

Must  take  “real”  process  behavior  into  consideration 
before  making  statistical  Inferences  about  its  performance. 

•  What  is  the  normal  or  inherent  process  variation? 

•  What  differentiates  inherent  from  anomalous  variation? 

•  What  is  causing  the  anomalous  variation? 

•  Why  is  the  anomalous  variation  occurring? 
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Methods  and  tools  are  needed  to  measure  and 
analyze  process  behavior  so  that  Inductive 
Inferences  about  the  process  performance  can 
be  supported. 
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Process  Performance 

Process  performance  is  behavior  that  can  be  described  or 
measured  using  attributes  of 

•  process  operation  or  execution 

•  resultant  products  or  services 

Process  performance  measures  answer  this  question: 
“How  is  the  process  performing  with  respect  to 
quality,  quantity,  effort  (cost),  and  time?” 

Process  performance  analysis  answers  this  question: 
“Why  is  the  process  behaving  as  it  is?” 
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Process  Variation 

Shewhart’s  notion  of  dividing  variation  into  two  types: 

1 .  Common  cause  variation 

•  variation  in  process  performance  due  to  normal  or 
inherent  interaction  among  process  components 
(people,  machines,  material,  environment,  and 
methods) 

2.  Assignable  cause  (special)  variation 

•  variation  in  process  performance  due  to  events  that  are 
not  part  of  the  normal  process 

•  represents  sudden  or  persistent  abnormal  changes  to 
one  or  more  of  the  process  components 
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Controlled  (Predictable)  Variation 


Process  behavior  measured  at  times  tl ,  t2,  t3,  and  t4 
All  measurements 

•  have  same  central  tendency  and  dispersion 


Distribution 
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Uncontrolled  (Unpredictable)  Variation 


Process  behavior  measured  at  times  t1 ,  t2,  t3,  and  t4 
Not  all  measurements 

•  have  same  central  tendency  and  dispersion 

•  fall  within  the  same  limits 
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Flagging  Variation 

Control  charts 

•  a  diagnostic  mechanism  to  determine  if  a  process  is  stable  and  to 
flag  variation  that  requires  causal  analysis. 

•  when  at  desired  performance,  they  are  also  a  control  mechanism 


SIX  SIGMA  ADVANTAGE 


The  Third  IV9/S  * 


/ 

Specification 

Limits 


CL  +  3a 


CL-Scj 


-  Upper  Control  Limit  (UCL) 
Mean  or  Center  Line  (CL) 


- —  Lower  Control  Limit  (LCL) 


Event  Time  or  Sequence 


Other  popular  methods  that  flag  variation  and  launch  causal  analysis 

•  distributions 

•  boxplots  (also  relative  to  time) 
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Patterns 


Rapid  Shift  in  L 
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Root  Cause  Terms 

Problem 

•  any  deviation  from  the  standard,  expected,  or  desired 

•  that  which  is  outside  the  accepted  tolerance,  norm,  benchmark 

Symptom  or  direct  cause 

•  observable  indicator,  cue,  or  event  that  flags  the  existence  of  a 
problem 

Theory  or  hypothesis 

•  unproven  assertion  about  reasons  for  the  problems  &  symptoms 

Cause 

•  event,  factor,  or  circumstance  that  has  been  demonstrated  to 
produce  the  problem  or  deviation 

Root  cause 

•  the  cause  that  can  be  turned  on  &  off  to  produce  the  problem, 
which  is  not  itself  an  interim  symptom  resulting  from  another  cause 
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Finding  Root  Cause:  Why  Care? 

Applying  corrective  action  or  improvement  to  something  other 
than  root  cause  is  unlikely  to  result  in  sustained  improvement. 

A  brief  scenario: 

•  A  project  is  half  complete  and  has  overrun  cost  by  25%. 

•  The  project  team  replans  and  negotiates  an  adjusted  cost  with 
the  customer.  No  other  actions  are  taken 

What  would  you  expect  to  happen  if  the  root  cause  is 

•  constant  rotation  of  team  members  on/off  project 

•  increased  cost  of  purchased  materials 

-  with  all  purchases  now  complete 

What  is  the  possible  impact  for  future  projects?  For  the 
business? 
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Diagnosing  Root  Cause 

Key  Steps 

•  generate  and  organize  hypotheses 

-  additional  data  exploration,  characterization  as  needed 

•  select  hypotheses  to  test 

•  test  and  evaluate 

Generating  and  organizing  lists  of  hypotheses 

•  5  whys  (just  what  it  says:  keep  asking  “why”) 

•  Brainstorming 

•  PSM  Performance  Analysis  Model 

•  diagrams:  cause  &  effect,  affinity,  tree,  interrelationship 
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Diagnosing  Root  Cause 


Testing  hypotheses 

•  historical  review 

•  additional  data  collection 

•  process  decomposition 

•  experiments 

Evaluating  tests 

•  7  basic  tools 

•  capability  analysis 

•  means-comparison  tests 

•  diagramming  techniques 

•  failure  modes  and  effects  analysis 
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Root  Cause  Tips 

Play  detective.  Suspect  everything! 

Evaluate  paired  comparisons 

•  characteristics  of  a  “good  data  point”  vs. 

•  characteristics  of  a  “bad  data  point” 

Or  stratify  the  data  into  “good”  and  “bad”  subsets  and  evaluate. 

For  sporadic  problems,  troubleshoot  in  real  time  if  possible. 

•  memories  fade  quickly 


Frequently  used  indicators  for  symptoms 

•  frequency 

•  severity  or  impact 


If  you  can  turn  the  problem  on  and  off, 
you  probably  have  found  root  cause! 
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Outline 

Context:  The  Value  Proposition 
Approach  to  building  integrated  training 
A  roadmap  for  performance-driven  improvement 
Roadmap  connections  to  CMMI 
The  roadmap  in  practice:  case  example 
The  roadmap  in  practice:  mini  lectures  &  practice 

o  Summary 

•  Illustrative  summary 

•  Key  points 
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Composite  Project  Illustration 

—  Problem  and  goal  statement  (Y): 

•  maximum  latent  defects  released 

•  minimum  mean  time  between  failure  in  the  field 
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•  Problem  &  goal 
statements 

•  Define  boundaries 

•  Process  maps 

•  “Management  by  fact” 


•  Discovery:  paretos,  histograms,  distributions,  c&e 

•  Understanding:  root  cause,  critical  factors 

•  Improvement:  adjust  critical  factors,  redesign 

•  Performance:  on  target,  with  desired  variation 

Y  =  f(defect  profile,  yield) 

=  f(review  rate,  method,  complexity . ) 
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Defined  Processes 
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[LMCO  02] 
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Project  Boundaries 
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Process  Map 


Design 

r 

Code 

r 

Compile 

- > 

Unit 

Test 

O  Requirements 
*  Estimate 
O  Concept  design 


O  Resources 

♦  Code  Stds 

♦  LOC  counter 
t  Interruptions 


O  Code 


O  Executable  Code 

•  Test  Plan,  Technique 

•  Operational  Profiles 


> 


•  Detailed  Design 

•  Test  cases 

•  Complexity 

•  Data:  Design  Review 
defects,  Fix  time. 
Phase  duration 


•  Code 

•  Data:  Defects, 
Fix  time.  Defect 
Injection  Phase, 
Phase  duration 


•  Executable 
Code 

•  Data:  Defects, 
Fix  time.  Defect 
Injection  Phase, 
Phase  duration 


•  Functional 
Code 

•  Data:  Defects, 
Fix  time.  Defect 
Injection  Phase, 
Phase  duration 


A 

Inspection 

O 

Critical  Inputs 

*  Standard  Procedure 

O 

Rework 

t 

Noise 

•  Control  Knobs 

j 
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Drilldown  to  Inspection  Process 

What  are  the  sources  of  variation?  The  controi  knobs? 


SIX  SIGMA  ADVANTAGE 
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O  Artifacts  to 
inspect 

•,  f  Artifact  size 

•  Reviewers 

*  Data  repository 


O  Review  Rate 
O ,  ♦  Checklists 
•,  ♦  Inspection 
method,  procedure 
f  Proficiency 
t  Taxonomy 
interpretations 


Critical  Inputs 

t 

Noise 

Standard  Procedure 

• 

Control  Knobs 

V 

J 

What  would 
you  list? 


What  would 
you  list? 


Defect  Log  •  Direct  Cause  •  Corrective 
^Record  of  plan  •  Root  Cause  Action 


c 


Data  feed  DMAiC 
project  process 
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Collecting  basic  data 

Refining  processes 

•  inspections 

Improving  using 

•  cause  &  effect  matrix 

•  pareto  analysis 

Leads  to 

•  injecting  fewer  defects 

•  detecting  defects  earlier 

•  removing  them  efficiently 

•  process  stability 


Initial  Defect  Density 


Design  Code 
Review  Review 


Post-Improvement  Defect  Density 


Design  Code 


Review  Review 
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Rayleigh  Distribution  as  Control 
Mechanism 


Error  Discovery  Data  and  Rayleigh  Fitted  Histograms 


Preliminary  Detailed  Code  &  Unit  Integration  System  Test  Deployment 
Design  Design  Test 
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Forecasting  Defects  and  Repair  Cost^lSy  Phase 

Pre-reiease  and  post-reiease  defect  counts  can  drive  further  modeis  to  forecast  defects  and  their  repair 
costs  over  time,  by  deveiopment  phase: 
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Size  and 
Defect  Count 
Estimates 

(From  Forecast  Model) 


Defect  Analysis  Scorecard 

To-Be  Prediction 

© 2002  Six  Sigma  Advantage  Title! 


Enter  the  yellow  fields:  | _ | 

Size 

Size  units  (e.g.  SLOG,  KLOC,  Fpts) 

Total  Defects 

Defect  Source  Insertion  %s 

Phase  Containment  Effectiveness  (errors) 

Defect  Containment  Effectiveness  (upstream  defects) 
Hours  to  fix  one  defect 
Hourly  labor  rate  (loaded) 


Defect  Source  Distribution 


Total  Defects> 


Origin 

Phase 


Code  size 

Size  units 

1,000 

FP 

1,050 

Insertion 
%  C 

Reqts 

22% 

Design 

35% 

Code 

43% 

Test 

Found-in  Phase 

Reqts 

Design 

Code 

Test 

Post-Release 

127 

52 

29 

15 

8 

221 

81 

43 

23 

262 

123 

66 

Enter  Hours  to  Fix 
One  Defect 

for  each  Insertion 
and  find  category 


Hours  to  Fix  One 
Defect 


Totai  Fix  Hours 


Loaded  Defect  Fix 
Costs 


Reqts 


Design 


Total  Containment  Effectivenes 
(TCE) 
per  phase 

96%  Reqts 

94%  Design 

85%  Code 

_ Test _ 

90.7% I  <TCE 


Totals 


Predicted 
Defects  by 


Phase 


Defect 

Repair 

Costs 


Totals  $  15,119 


$  32,136  $  68,734 


Project 

Cost 

Rollup 


Hourly  labor  rate 
(loaded) 


100.00 


Cost  per  FP  : 


Person-months 


Cost  of  Effort 


Cost  per  Released  Defect  $ 


1,317,188 


Fix  Cost  °/ 


384,128 


384.13 


Rework  cost 
is  29%  high  ? 
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From  single  process  to  systemic  view: 
Cause-Effect  Model  Using  Bayesian  Modeling 

III  II 

Project  System  System  Contract  book  Design 
Initiation  Requirements  Requirements  Baseline  &  Readiness 
Baselined  Allocated  Approved 


I 

System 

Test 

Readiness 


Ready  for  Ready  for  Volume 

Field  Test  Controlled  Deployment 

Introduction 


[Stoddard  02] 
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Beyond  Control  Charts:  Prediction 
Models 

Bivariate  Rt  of  Actuais  By  Predicted 

I  60 

w  50 

^  40 

o  30 
< 

20 

10  H" 

^  0  10  20  30  40  50  60 

Predicted 


Actual  field  defects  =  f(CASRE  predicted  defects) 

CASRE  predicted  defects  =  f(weekiy  arrivai  rate  of  SW  faiiures, 
weekiy  test  intensity  measures) 

$3M/year  savings  from  premature  SW  reieases 

[Stoddard  02],  *CASRE  =  Computer  Aided  Software  Reliability  Estimation 
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Summary  -  Key  Points 

Value  Proposition 

•  It  is  a  multi-initiative  world 

•  Measurement  &  analysis  is  a  common  root  and  an  integrating 
platform 

Approaches  to  Integration 

•  Training  courses  are  a  mechanism  to  make  connections 

DMAIC  &  CMMI  relationships 

•  DMAIC  and  CMMI  are  different  and  synergistic 

•  Relationships  exist  at  the  PA,  goal,  practice  and  toolkit  level 

DMAIC  Execution 

•  Performance  driven  improvement 

•  Enables  CMMI  subprocess  selection  to  be  informed  by 
business  priorities,  baselines,  problem  statements  and  causal 
analysis 

•  Propels  you  to  high  maturity  and/or  high  capability 
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A  Base  Architecture 

-  Connecting  all  the  Improvement  Models 


[Kawakita],  [Shiba] 
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Contact  Information 


If  you  would  like  copies  of  our  slides 

•  Add  your  name  to  our  signup  sheet,  or 

•  Send  an  email  to  Debra  Morrison,  dtm@sei.cmu.edu. 

-  Put  “SEPG  Tutorial”  in  the  subject  line. 

For  questions  or  other  information: 

Jeannine  Siviy 

Software  Engineering  Institute 
imsiviv@sei.cmu.edu 

Dave  Hallowell 

Six  Sigma  Advantage 

dhallowell@SixSiama-Advantaae.com 
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Dave  Zubrow 

Software  Engineering  Institute 
dz@sei.cmu.edu 
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